Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 19th June 2019, 11:03   #1221  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Quote:
Originally Posted by Meloware View Post
Are there any significant changes suggested in the current 8mm script, when processing frames of varying size and film formats?

My 16mm frame sequence captures are processed at a resolution of 2604x1738px. What operations (denoise, stabilize, sharpen, etc) require special consideration when taking into account frame size?
There's no need for such a high resolution. I have wonderful results with 16mm film, captured at 1388x1032px. The quality of the capturing is more important than the size.

To speed up the script, I even downscale the original capturing to 1024x768px, then run the script, then upscale it to full HD. (Pillarbox to keep 4:3) It runs at 5fps in Avisynth+ with MT enabled.

PS: you are doing a great job on your website.

Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest
videoFred is offline   Reply With Quote
Old 19th June 2019, 11:05   #1222  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Quote:
Originally Posted by bassquake View Post
[*]Crop + Resize + Stabilise + Dirt Removal + Denoise & Sharpen + Gammac + Autolevels: ~0.4fps[/LIST]
I could probably live with 2fps but Autolevels seems to cause it to choke.
Why Autolevels if you are already use Gammac?

Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest
videoFred is offline   Reply With Quote
Old 19th June 2019, 11:29   #1223  |  Link
bassquake
Registered User
 
Join Date: Jan 2007
Posts: 45
Quote:
Originally Posted by videoFred View Post
Why Autolevels if you are already use Gammac?

Fred.
Because it was in John Meyers original script. I assumed was needed.
bassquake is offline   Reply With Quote
Old 19th June 2019, 12:59   #1224  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Quote:
Originally Posted by bassquake View Post
Because it was in John Meyers original script. I assumed was needed.
I see.... you can try it without Autolevels and see what you get.


Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest
videoFred is offline   Reply With Quote
Old 19th June 2019, 13:02   #1225  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Never tried this myself but this might help to speed up:
https://forum.doom9.org/showthread.p...57#post1846757

Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest
videoFred is offline   Reply With Quote
Old 19th June 2019, 14:24   #1226  |  Link
bassquake
Registered User
 
Join Date: Jan 2007
Posts: 45
Quote:
Originally Posted by videoFred View Post
I see.... you can try it without Autolevels and see what you get.
Fred.
As mentioned earlier, I get about 2.8fps with Autolevels off. If it's not needed then Ill permanently remove it.

Quote:
Never tried this myself but this might help to speed up:
https://forum.doom9.org/showthread.p...57#post1846757
Fred.
Thanks, will have a look.
bassquake is offline   Reply With Quote
Old 19th June 2019, 16:01   #1227  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
I forgot to take Autolevels out once I added GamMac. Autolevels never worked for me and I never, ever used it.

Also, I second Fred's recommendation to use a lower resolution.

To support Fred's advice about using lower resolution, I should post a comparison to my transfer of some 1930s German B&W film which I transferred for a friend a dozen years ago. Since the film involves Nazis, I have never posted it publicly. However, the following link shows some non-controversial film from the same source (over 200 400' reels of film were smuggled out):

https://www.youtube.com/watch?v=qpM9jpmYMt4

Because of the historical significance of some of the reels, when the Smithsonian Channel found out about it, they wanted to use it as part of a documentary. Long story made short, they insisted on having it transferred by a professional lab in L.A. that uses a Cintel/Spirit scanner. I had them send the scans to me before I sent the results to the Smithsonian, and I did a comparison of my transfer vs. theirs.

The punch line to the story is that yes, they did do a better job than my camcorder capture off my modified, shutterless Eiki projector, but the differences in resolution between their 4K scan and my 720x480 (down res'd in my Sony FX1 HD camcorder) were extremely small, even though this was 16mm film and even though it was taken by a man who owned a German movie camera factory (i.e., he knew what he was doing). The place where they were better, although by less than I expected, was in the shadow details.

If the film had been professional film, shot on high-end cameras with a $50,000 lens, and developed in a pro lab, I think the resolution differences would have been obvious and substantial. However, when you are dealing with amateur film, shot on a $100 camera and developed at the local drug store, there just isn't a lot there for the extra HD resolution to preserve.

Do some tests yourself to make sure this is the case with your material, but I suspect that you won't see any difference whatsoever if you follow Fred's advice, and you'll end up saving yourself a huge amount of time by using a lower resolution.
johnmeyer is offline   Reply With Quote
Old 19th June 2019, 16:32   #1228  |  Link
Meloware
Registered User
 
Join Date: Dec 2018
Location: Massachusetts USA
Posts: 25
@johnmeyer and @videoFred's comments on resolution are well taken and worthy of my own study. Thanks..
Everyone's help has contributed real and rapid progress in my effort! There are obvious concepts, in Avisynth, which have alluded me for years, that I am now gaining a grasp of! I plan on standardizing my script and source files, and then benchmark them on a number of applications (AvsPmod, VirtualDub, and ffmpeg64). Then, there will be many questions!

I mentioned my system as having a 32 core Threadripper, with 64GB ram. My source and destination files are to and from SSDs. I supposedly have 64 threads of processing available, yet any stable run of my script uses only 3% of my CPU, at best. The script's “threads” variable actually slows down processing, as I increase the number, and “SetMemoryMax” has very little beneficial effect. I have never seen more than 18% of my fast RAM ever used. I'd really like to light up my system, but I will have to ask the meaningful questions later.

I would like to mention a trick, which occurred to me when I was using 32 bit AviSynth 2.6 with Vdub.

There is an important advantage in rendering your films to frame sequences. I have yet to run any script, which alone is able to fully utilize the processing power of my Threadripper system, or even my 10 year old i7 machine. I got around that problem by launching multiple instances of my host program (VirtualDub) and then pointing each instance to it's own range of frames to process. Five instances of the program can process 5 times faster, than a single instance! This saved me from waiting a week to process a 25,000 frame film on my i7! My Threadripper is happy to assign each instance it's own memory and processing thread resources. The only problem is, that it takes time to setup and manage 50 or 60 instances of VirtualDub. Still, it is worth the effort if there is no other option. Once all the individual frames are processed, it is a simple matter to pass it to Blender (for the final touches) or transcode it into any playable video format you wish.

This trick will work with other, off topic applications. It is even possible to to run multiple instances of these bone-head commercial programs, which prohibit multiple instances. In that case, just create additional users on your machine, and then launch each instance as a separate user.

I will try and report back some solid benchmarks, later. Thanks again, to all.
Meloware is offline   Reply With Quote
Old 19th June 2019, 16:45   #1229  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by Meloware View Post
The script's “threads” variable actually slows down processing
If you're referring to bassquake's script above, the 'thread' variable does nothing. Post your complete script, the Avisynth version you're using and the DLL versions.
If you want to do all that in one go, use AVSMeter with this command line:
"AVSMeter script.avs -info -log". This will produce a log file in the script's directory. Post that file.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 19th June 2019, 16:50   #1230  |  Link
Meloware
Registered User
 
Join Date: Dec 2018
Location: Massachusetts USA
Posts: 25
Quote:
Originally Posted by Groucho2004 View Post
If you're referring to bassquake's script above, the 'thread' variable does nothing. Post your complete script, the Avisynth version you're using and the DLL versions.
If you want to do all that in one go, use AVSMeter with this command line:
"AVSMeter script.avs -info -log". This will produce a log file in the script's directory. Post that file.
I will do that. Yes, this was with @bassquake's version, posted a few days ago. First, l need some time to get my 'day job' out of the way and then produce a version I can benchmark. I will be certain to include the source info and AVSMeter report.
Meloware is offline   Reply With Quote
Old 19th June 2019, 19:32   #1231  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Quote:
any stable run of my script uses only 3% of my CPU
Something definitely is not right. Even without MT enabled, you should be getting better utilization than that.
johnmeyer is offline   Reply With Quote
Old 20th June 2019, 02:23   #1232  |  Link
Meloware
Registered User
 
Join Date: Dec 2018
Location: Massachusetts USA
Posts: 25
Here is a copy of my AVSMeter log. The script is too long to include in this post, but could be re-posted if asked.

Source files were jpg frames 2604 x 1738 resolution and the output frames are written as 1640 x 1280 and are intended to be exported to Blender 3D for final tweaking and resizing.

Performance with these source images are a miserable .1 fps. Strangely, another run I made with source frames cropped and resized to 2260 x 1738 ran at .8fps.

CPU performance was around 3%. Total Memory consumption is below 12%. I did experiment with using ffmpeg64 and 14 CMD prompt instances. The CPU utilization was then 35% and memory at 74%. My fans spun up the system then produced close to 700 frames per minute. Is something obviously wrong? What is keeping the 64 threads from kicking in?

Code:
C:\software\AVSMeter292\AVSMeter292>AVSMeter64 avsinfo

AVSMeter 2.9.2 (x64) - Copyright (c) 2012-2019, Groucho2004

VersionString:              AviSynth+ 0.1 (r2772, MT, x86_64)
VersionNumber:              2.60
File / Product version:     0.1.0.0 / 0.1.0.0
Interface Version:          6
Multi-threading support:    Yes
Avisynth.dll location:      C:\WINDOWS\SYSTEM32\avisynth.dll
Avisynth.dll time stamp:    2018-12-20, 17:55:16 (UTC)
PluginDir2_5 (HKLM, x64):   C:\Program Files (x86)\AviSynth+\plugins64
PluginDir+   (HKLM, x64):   C:\Program Files (x86)\AviSynth+\plugins64+


[CPP 2.5 Plugins (64 Bit)]
C:\Program Files (x86)\AviSynth+\plugins64+\warpsharp.dll  [2011-06-15]

[CPP 2.6 Plugins (64 Bit)]
C:\Program Files (x86)\AviSynth+\plugins64+\AutoLevels_x64.dll  [0.10.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\ConvertStacked.dll  [2018-12-20]
C:\Program Files (x86)\AviSynth+\plugins64+\Deflicker.dll  [0.6.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\DePan.dll  [2.13.1.4]
C:\Program Files (x86)\AviSynth+\plugins64+\DePanEstimate.dll  [2.10.0.3]
C:\Program Files (x86)\AviSynth+\plugins64+\DirectShowSource.dll  [2018-12-20]
C:\Program Files (x86)\AviSynth+\plugins64+\ffms2.dll  [2016-12-29]
C:\Program Files (x86)\AviSynth+\plugins64+\FredAverage_x64.dll  [0.3.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\GamMac_x64.dll  [1.10.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\ImageSeq.dll  [2018-12-20]
C:\Program Files (x86)\AviSynth+\plugins64+\masktools2.dll  [2.2.18.0]
C:\Program Files (x86)\AviSynth+\plugins64+\mvtools2.dll  [2.7.41.0]
C:\Program Files (x86)\AviSynth+\plugins64+\nnedi3.dll  [0.9.4.53]
C:\Program Files (x86)\AviSynth+\plugins64+\RemoveDirt.dll  [0.9.2.0]
C:\Program Files (x86)\AviSynth+\plugins64+\RgTools.dll  [0.97.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\Shibatch.dll  [2018-12-20]
C:\Program Files (x86)\AviSynth+\plugins64+\TimeStretch.dll  [2018-12-20]
C:\Program Files (x86)\AviSynth+\plugins64+\VDubFilter.dll  [2018-12-20]

[Scripts (AVSI)]
C:\Program Files (x86)\AviSynth+\plugins64+\colors_rgb.avsi  [2016-07-05]
C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC.avsi  [2019-06-17]
C:\Program Files (x86)\AviSynth+\plugins64+\smdegrain.avsi  [2019-06-17]

[Uncategorized DLLs (64 Bit)]
C:\Program Files (x86)\AviSynth+\plugins64+\libfftw3f-3.dll  [2019-06-17]

[Uncategorized files]
C:\Program Files (x86)\AviSynth+\plugins64+\colors_rgb.txt  [2016-07-05]



[Plugin errors/warnings]
______________________________________________________________________________________________________

Cannot load file 'C:/Program Files (x86)/AviSynth+/plugins64+/nnedi3.dll'. Platform returned code 126:
The specified module could not be found.

Dependencies that could not be loaded:
  libmmd.dll
______________________________________________________________________________________________________


Log file created with:      AVSMeter 2.9.2 (x64)
Script file:                C:\Program Files (x86)\AviSynth+\scripts\Meloware.avs

[OS/Hardware info]
Operating system:           Windows 10 (x64) (Build 17763)

CPU:                        AMD Ryzen Threadripper 2990WX 32-Core Processor / Threadripper (Colfax)
                            MMX, MMXEXT, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, SSE4A, FMA3, AVX, AVX2
                            32 physical cores / 64 logical cores


[Avisynth info]
VersionString:              AviSynth+ 0.1 (r2772, MT, x86_64)
VersionNumber:              2.60
File / Product version:     0.1.0.0 / 0.1.0.0
Interface Version:          6
Multi-threading support:    Yes
Avisynth.dll location:      C:\WINDOWS\SYSTEM32\avisynth.dll
Avisynth.dll time stamp:    2018-12-20, 17:55:16 (UTC)
PluginDir2_5 (HKLM, x64):   C:\Program Files (x86)\AviSynth+\plugins64
PluginDir+   (HKLM, x64):   C:\Program Files (x86)\AviSynth+\plugins64+


[Clip info]
Number of frames:                 1586
Length (hh:mm:ss.ms):     00:01:06.083
Frame width:                      1640
Frame height:                     1280
Framerate:                      24.000 (24/1)
Colorspace:                       YV12
Audio channels:                    n/a
Audio bits/sample:                 n/a
Audio sample rate:                 n/a
Audio samples:                     n/a
Meloware is offline   Reply With Quote
Old 20th June 2019, 02:40   #1233  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by Meloware View Post
Is something obviously wrong? What is keeping the 64 threads from kicking in?
Need to see the script. You can post it on pastebin.com.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 20th June 2019, 03:49   #1234  |  Link
Meloware
Registered User
 
Join Date: Dec 2018
Location: Massachusetts USA
Posts: 25
Here is the script used in my last post:

Code:
# John Meyers Script
# Use AVISynth version 2.60
#
# This script version is being evaluated by Meloware (June 19, 2019 ver.1) for optimal performance 
# on a AMD 32 core Threadripper 2990WX system with 64GB ram - Windows 10
# The source files for this experiment are 2604 x 1738 jpg images captured and resized from a Sony A6300 mirrorless camera.
# The output frames produced by this script would normally then be exported to Blender 3D's compositor, for final tweaks and resizing
# into 1440 x 1080 pixel frames for final transcoding into a usable video.
# Many examples of my restored 16mm films are at http://industrialhistory.org/education/videos/
#
#VIDEO FILE
#----------------------------------------------------------------------------------------------------------------------------

SetFilterMTMode("DEFAULT_MT_MODE", 2)

#Change the following line to point to your video file
film="J:\film\Prestone\source\p-%04d.jpg"

##### Destination files (if option "C1" is used and ImageWriter writes individual frames to a defined directory
output= "J:\film\Prestone\stab\p-%04d.jpg"


#GENERAL PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="result1"                                               #specify the wanted output here; valid results are "result1" through "result4" and "resultS1" through "resultS4" for before/after

trim_begin=1  trim_end= 1586

play_speed=18                                                  #play speed (8mm=16; Super8=18; 16mm sound=24)

fps = 24







#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation=0.9                                                  #for all outputs
gamma=1.0                                                       #for all outputs 
blue= -0  red=-0                                                #manual color adjustment, when returning result3 & result4. Values can be positive or negative
black_level=0  white_level=255 output_black=0  output_white=255 #manual levels, when returning result2 & result4
 

#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=250  CTop=100  CRight=300  CBottom=10                         #crop values after Depan and before final resizing 
W=1640  H=1280                                                     #final size after cropping 
bord_left=0        bord_top=0     bord_right=0     bord_bot=0    #720p= borders 150
in_bord_left=0  in_bord_top=0  in_bord_right=0  in_bord_bot=0    #Borders around input that must be removed


#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=20                                                      #maximum values for the stabiliser (in pixels) 20 is a good start value 
maxstabV=40
est_left=760   est_top=380  est_right=500  est_bottom=350             #crop and contast values for special Estimate clip
est_cont=1.6

#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
#Fastest parameters
#denoising_strength=600                     #denoising level of first denoiser: MVDegrain() 
#denoising_strength=300                     #denoising level of first denoiser: MVDegrain() 
#block_size=   16                           #block size of MVDegrain
#block_size_v= 16
#block_over=    8                           #block overlapping of MVDegrainMulti()

#Best compromise between speed and quality
denoising_strength= 0                     #denoising level of first denoiser: MDegrain() 
block_size=    8                            #block size of MVDegrain
block_size_v=  8
block_over=    4                            #block overlapping of MVDegrainMulti()

dirt_strength=20                            #sets amount of dirt removal (big spots)


#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
PRE_sharp_ness= 30   PRE_radi_us= 3              #presharpening (UnsharpMask) just after first denoising   
Sharp_Strength= 0.1


#AUTO LEVELS PARAMETER
#--------------------------------------------------------------------------------------------------------------------------------
X=4   #X  is a special parameter for reducing the autolevels effect on the whites
X2=16  #X2 is a special parameter for reducing the autolevels effect on the blacks 


#NUMBER OF THREADS
#--------------------------------------------------------------------------------------------------------------------------------
threads=5  #####What does this do?


# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================

# Using AVISynth version 2.60, build August 28, 2012 (Ben-Rudiak-Gould, etc.)
# Using VirtualDub 1.8.6

#Change the following (lower or higher) if you have stability problems with multi-threading
SetMemoryMax(800)  ## What does this do

#Load plugins explicitly
LoadPlugin("../plugins64+/mvtools2.dll")
LoadPlugin("../plugins64+/autolevels_x64.dll")
LoadPlugin("../plugins64+/Deflicker.dll")
Loadplugin("../plugins64+/Depan.dll")
LoadPlugin("../plugins64+/DepanEstimate.dll")
Loadplugin("../plugins64+/masktools2.dll")
Loadplugin("../plugins64+/RemoveDirt.dll")    
Loadplugin("../plugins64+/RgTools.dll")
Loadplugin("../plugins64+/warpsharp.dll")

#Use the following for alternative frame interpolation
#loadplugin("svpflow1.dll")        #Version 1.2.1.0     5/29/2012
#loadplugin("svpflow2.dll")        #Version 1.3.1.0     6/02/2012 
#Import("InterFrame2.avsi")        #Version 2.1.0       6/27/2012

#Open and crop the video
#Remove all setmtmode statements (there are four in this script) if not using multi-threaded (MT) AVISynth
#setmtmode(5,threads)
source1= ImageSource(file=film, trim_begin, trim_end).converttoYV12()           #killaudio() improves stability (in my experience -- others disagree)
cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot)    #temporarily remove any black borders on input video
#setmtmode(2)

#cropped_source=filldrops(cropped_source)     #Use this when removing bad frames that have been removed by duplicating previous frame
                                              #(see notes in function at end of script)

#STABILIZING
#....................................................................................................................................................................
stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)

#Alternative line that includes deflicker. I find that this usually is not needed.
#stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15).deflicker()


#DENOISING
#...................................................................................................................................................................

#Remove dirt
input_to_removedirt=stab.crop(CLeft,CTop,-CRight,-CBottom)
stabcrop=RemoveDirtMC(input_to_removedirt,dirt_strength,false)

#Reduce grain
prefiltered = RemoveGrain(stabcrop,2)
superfilt =   MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super=        MSuper(stabcrop, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4)   ? block_size/2 : 4
halfoverlap= (block_over>2)   ? block_over/2 : 2

bvec1 =  MAnalyse(superfilt, isb = true,  delta = 1, blksize=block_size, overlap=block_over,dct=0)
bvec1 =  MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100) 

fvec1 =  MAnalyse(super, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
fvec1 =  MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 =  MAnalyse(super, isb = true,  delta = 2, blksize=block_size, overlap=block_over,dct=0)
bvec2 =  MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 =  MAnalyse(super, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
fvec2 =  MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)


#SHARPENING
#...................................................................................................................................................................
sharp1=denoised.sharpen(Sharp_Strength)	
PreBorderFrame = sharp1.Lanczos4Resize(W - bord_left - in_bord_left - bord_right - in_bord_right, H - bord_top - in_bord_top - bord_bot - in_bord_bot)


#FRAME INTERPOLATION (optional)
#...................................................................................................................................................................

#Interpolate frames
#Use this for progressive output -- example shows 29.97 progressive
#PreBorderFrame=InterFrame(NewNum=30000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads)

#Use this for interlaced output -- example shows 29.97 interlaced
#PreBorderFrame=InterFrame(NewNum=60000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads).SeparateFields().SelectEvery(4, 0, 3).Weave()


#RESULT1: AUTOLEVELS,AUTOWHITE
#......................................................................................................................................................................
#SetMTMode(5)         #Turn off SetMTMode for Autolevels
result1= PreBorderFrame.coloryuv(autowhite=true).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT3: AUTOLEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result3= PreBorderFrame.coloryuv(off_U=blue,off_V=red).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)
#SetMTMode(2)        #Re-enable SetMTMode after Autolevels

#RESULT2: MANUAL LEVELS, AUTOWHITE
#......................................................................................................................................................................
result2= PreBorderFrame.levels(black_level,gamma,white_level,0,255).coloryuv(autowhite=true).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT4: MANUAL LEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result4= PreBorderFrame.coloryuv(off_U=blue,off_V=red).levels(black_level,gamma,white_level,0,255).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)


#PARAMETERS FOR THE COMPARISONS
#.....................................................................................................................................................................
W2= W+bord_left+bord_right
H2= H+bord_top+bord_bot
source4=Lanczos4Resize(source1,W2,H2)


#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"autolevels, autowhite",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"autowhite, manual levels correction",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"autolevels, manual color correction",size=28,align=2))
resultS4= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result4,"manual colors and levels correction",size=28,align=2))

resultC1= result1.ConverttoRGB24().ImageWriter(file= output,start = 1, type = "jpg", info = true)

#Finish and go home
Eval(result)


# END SCRIPT, BEGIN FUNCTIONS
#=================================================================================================================================


#REMOVE DIRT FUNCTION
#......................................................................................................................................................................
function RemoveDirt(clip input, int limit, bool _grey)
{
  clensed=input.Clense(grey=_grey, cache=4)
  alt=input.RemoveGrain(2)
  return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

  # Alternative settings
  # return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
  # return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

#This function provides motion compensation for the remove dirt function, thus improving quality.
function RemoveDirtMC(clip,int "limit", bool "_grey")
{
  _grey=default(_grey, false)
  limit = default(limit,6)
 
  prefiltered = RemoveGrain(clip,2)
  superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
  super=MSuper(clip, hpad=32, vpad=32,pel=2)

  bvec = MAnalyse(superfilt,isb=true,  blksize=16, overlap=2,delta=1, truemotion=true)
  fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

  bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
  fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

  backw = MFlow(clip,super,bvec_re)
  forw  = MFlow(clip,super,fvec_re)

  clp=interleave(forw,clip,backw)
  clp=clp.RemoveDirt(limit,_grey)
  clp=clp.SelectEvery(3,1)
  return clp
}

# The following function will remove near duplicates ("0.1") or exact duplicates (change to "0.0"). 
# It replaces these duplicates with a motion estimated frame. It works really well (IMHO).
#
# Use: In your video editor, replace any single bad frame (burned frame, jump, missing frame from splice, etc.) with a duplicate of the 
# previous frame. Then, include a call to this function (see script above, where it is commented out).
#
# Here's an example of how this function can be use:
#
# http://www.youtube.com/watch?v=uzMFodrGHDs
#

function filldrops (clip c)
{
  super=MSuper(c,pel=2)
  vfe=manalyse(super,truemotion=true,isb=false,delta=1)
  vbe=manalyse(super,truemotion=true,isb=true,delta=1)
  filldrops = mflowinter(c,super,vbe,vfe,time=50)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
  return fixed
}

Prefetch(4)

#return last
Meloware is offline   Reply With Quote
Old 20th June 2019, 10:04   #1235  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by Meloware View Post
Here is the script used in my last post:

Code:
# John Meyers Script

SetFilterMTMode("DEFAULT_MT_MODE", 2)

#Change the following line to point to your video file
film="J:\film\Prestone\source\p-%04d.jpg"

##### Destination files (if option "C1" is used and ImageWriter writes individual frames to a defined directory
output= "J:\film\Prestone\stab\p-%04d.jpg"


#GENERAL PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="result1"                                               #specify the wanted output here; valid results are "result1" through "result4" and "resultS1" through "resultS4" for before/after

trim_begin=1  trim_end= 1586

play_speed=18                                                  #play speed (8mm=16; Super8=18; 16mm sound=24)

fps = 24


#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation=0.9                                                  #for all outputs
gamma=1.0                                                       #for all outputs 
blue= -0  red=-0                                                #manual color adjustment, when returning result3 & result4. Values can be positive or negative
black_level=0  white_level=255 output_black=0  output_white=255 #manual levels, when returning result2 & result4
 

#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=250  CTop=100  CRight=300  CBottom=10                         #crop values after Depan and before final resizing 
W=1640  H=1280                                                     #final size after cropping 
bord_left=0        bord_top=0     bord_right=0     bord_bot=0    #720p= borders 150
in_bord_left=0  in_bord_top=0  in_bord_right=0  in_bord_bot=0    #Borders around input that must be removed


#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=20                                                      #maximum values for the stabiliser (in pixels) 20 is a good start value 
maxstabV=40
est_left=760   est_top=380  est_right=500  est_bottom=350             #crop and contast values for special Estimate clip
est_cont=1.6

#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
#Fastest parameters
#denoising_strength=600                     #denoising level of first denoiser: MVDegrain() 
#denoising_strength=300                     #denoising level of first denoiser: MVDegrain() 
#block_size=   16                           #block size of MVDegrain
#block_size_v= 16
#block_over=    8                           #block overlapping of MVDegrainMulti()

#Best compromise between speed and quality
denoising_strength= 0                     #denoising level of first denoiser: MDegrain() 
block_size=    8                            #block size of MVDegrain
block_size_v=  8
block_over=    4                            #block overlapping of MVDegrainMulti()

dirt_strength=20                            #sets amount of dirt removal (big spots)


#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
PRE_sharp_ness= 30   PRE_radi_us= 3              #presharpening (UnsharpMask) just after first denoising   
Sharp_Strength= 0.1


#AUTO LEVELS PARAMETER
#--------------------------------------------------------------------------------------------------------------------------------
X=4   #X  is a special parameter for reducing the autolevels effect on the whites
X2=16  #X2 is a special parameter for reducing the autolevels effect on the blacks 


#NUMBER OF THREADS
#--------------------------------------------------------------------------------------------------------------------------------
threads=5  #####What does this do?


# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================

# Using AVISynth version 2.60, build August 28, 2012 (Ben-Rudiak-Gould, etc.)
# Using VirtualDub 1.8.6

#Change the following (lower or higher) if you have stability problems with multi-threading
SetMemoryMax(800)  ## What does this do

#Load plugins explicitly
LoadPlugin("../plugins64+/mvtools2.dll")
LoadPlugin("../plugins64+/autolevels_x64.dll")
LoadPlugin("../plugins64+/Deflicker.dll")
Loadplugin("../plugins64+/Depan.dll")
LoadPlugin("../plugins64+/DepanEstimate.dll")
Loadplugin("../plugins64+/masktools2.dll")
Loadplugin("../plugins64+/RemoveDirt.dll")    
Loadplugin("../plugins64+/RgTools.dll")
Loadplugin("../plugins64+/warpsharp.dll")

#Use the following for alternative frame interpolation
#loadplugin("svpflow1.dll")        #Version 1.2.1.0     5/29/2012
#loadplugin("svpflow2.dll")        #Version 1.3.1.0     6/02/2012 
#Import("InterFrame2.avsi")        #Version 2.1.0       6/27/2012

#Open and crop the video
#Remove all setmtmode statements (there are four in this script) if not using multi-threaded (MT) AVISynth
#setmtmode(5,threads)
source1= ImageSource(file=film, trim_begin, trim_end).converttoYV12()           #killaudio() improves stability (in my experience -- others disagree)
cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot)    #temporarily remove any black borders on input video
#setmtmode(2)

#cropped_source=filldrops(cropped_source)     #Use this when removing bad frames that have been removed by duplicating previous frame
                                              #(see notes in function at end of script)

#STABILIZING
#....................................................................................................................................................................
stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)

#Alternative line that includes deflicker. I find that this usually is not needed.
#stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15).deflicker()


#DENOISING
#...................................................................................................................................................................

#Remove dirt
input_to_removedirt=stab.crop(CLeft,CTop,-CRight,-CBottom)
stabcrop=RemoveDirtMC(input_to_removedirt,dirt_strength,false)

#Reduce grain
prefiltered = RemoveGrain(stabcrop,2)
superfilt =   MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super=        MSuper(stabcrop, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4)   ? block_size/2 : 4
halfoverlap= (block_over>2)   ? block_over/2 : 2

bvec1 =  MAnalyse(superfilt, isb = true,  delta = 1, blksize=block_size, overlap=block_over,dct=0)
bvec1 =  MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100) 

fvec1 =  MAnalyse(super, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
fvec1 =  MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 =  MAnalyse(super, isb = true,  delta = 2, blksize=block_size, overlap=block_over,dct=0)
bvec2 =  MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 =  MAnalyse(super, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
fvec2 =  MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)


#SHARPENING
#...................................................................................................................................................................
sharp1=denoised.sharpen(Sharp_Strength)	
PreBorderFrame = sharp1.Lanczos4Resize(W - bord_left - in_bord_left - bord_right - in_bord_right, H - bord_top - in_bord_top - bord_bot - in_bord_bot)


#FRAME INTERPOLATION (optional)
#...................................................................................................................................................................

#Interpolate frames
#Use this for progressive output -- example shows 29.97 progressive
#PreBorderFrame=InterFrame(NewNum=30000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads)

#Use this for interlaced output -- example shows 29.97 interlaced
#PreBorderFrame=InterFrame(NewNum=60000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads).SeparateFields().SelectEvery(4, 0, 3).Weave()


#RESULT1: AUTOLEVELS,AUTOWHITE
#......................................................................................................................................................................
#SetMTMode(5)         #Turn off SetMTMode for Autolevels
result1= PreBorderFrame.coloryuv(autowhite=true).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT3: AUTOLEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result3= PreBorderFrame.coloryuv(off_U=blue,off_V=red).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)
#SetMTMode(2)        #Re-enable SetMTMode after Autolevels

#RESULT2: MANUAL LEVELS, AUTOWHITE
#......................................................................................................................................................................
result2= PreBorderFrame.levels(black_level,gamma,white_level,0,255).coloryuv(autowhite=true).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT4: MANUAL LEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result4= PreBorderFrame.coloryuv(off_U=blue,off_V=red).levels(black_level,gamma,white_level,0,255).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)


#PARAMETERS FOR THE COMPARISONS
#.....................................................................................................................................................................
W2= W+bord_left+bord_right
H2= H+bord_top+bord_bot
source4=Lanczos4Resize(source1,W2,H2)


#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"autolevels, autowhite",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"autowhite, manual levels correction",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"autolevels, manual color correction",size=28,align=2))
resultS4= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result4,"manual colors and levels correction",size=28,align=2))

resultC1= result1.ConverttoRGB24().ImageWriter(file= output,start = 1, type = "jpg", info = true)

#Finish and go home
Eval(result)


# END SCRIPT, BEGIN FUNCTIONS
#=================================================================================================================================


#REMOVE DIRT FUNCTION
#......................................................................................................................................................................
function RemoveDirt(clip input, int limit, bool _grey)
{
  clensed=input.Clense(grey=_grey, cache=4)
  alt=input.RemoveGrain(2)
  return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

  # Alternative settings
  # return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
  # return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

#This function provides motion compensation for the remove dirt function, thus improving quality.
function RemoveDirtMC(clip,int "limit", bool "_grey")
{
  _grey=default(_grey, false)
  limit = default(limit,6)
 
  prefiltered = RemoveGrain(clip,2)
  superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
  super=MSuper(clip, hpad=32, vpad=32,pel=2)

  bvec = MAnalyse(superfilt,isb=true,  blksize=16, overlap=2,delta=1, truemotion=true)
  fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

  bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
  fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

  backw = MFlow(clip,super,bvec_re)
  forw  = MFlow(clip,super,fvec_re)

  clp=interleave(forw,clip,backw)
  clp=clp.RemoveDirt(limit,_grey)
  clp=clp.SelectEvery(3,1)
  return clp
}

function filldrops (clip c)
{
  super=MSuper(c,pel=2)
  vfe=manalyse(super,truemotion=true,isb=false,delta=1)
  vbe=manalyse(super,truemotion=true,isb=true,delta=1)
  filldrops = mflowinter(c,super,vbe,vfe,time=50)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
  return fixed
}

Prefetch(4)

#return last
Add these lines to the very top of your script:
Code:
SetMemoryMax(16000) #increase to 32000 or even 40000 if necessary.
threads = 16 #increase gradually up to 32 or 40. Any higher than that, the memory consumption will get close to your 64G (I think)
### Check speed, memory consumption and CPU usage with AVSMeter64 ###
Change the last line in your script ("Prefetch(4)") to this:
Code:
Prefetch(threads)
Also, put avstp.dll into your plugins64+ directory. It appears to be quite beneficial for this particular script.
__________________
Groucho's Avisynth Stuff

Last edited by Groucho2004; 20th June 2019 at 10:19.
Groucho2004 is offline   Reply With Quote
Old 20th June 2019, 10:37   #1236  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Quote:
Originally Posted by Meloware View Post
Source files were jpg frames 2604 x 1738 resolution and the output frames are written as 1640 x 1280 and are intended to be exported to Blender 3D for final tweaking and resizing.
Why Blender 3D? It can all be done in one Avisynth script. If you want sliders you can tweak the script for AvsPmod.

Can you upload an original untouched image sequence somewhere? 50 frames or so...

Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest
videoFred is offline   Reply With Quote
Old 20th June 2019, 15:34   #1237  |  Link
Meloware
Registered User
 
Join Date: Dec 2018
Location: Massachusetts USA
Posts: 25
Quote:
Originally Posted by videoFred View Post
Why Blender 3D? It can all be done in one Avisynth script. If you want sliders you can tweak the script for AvsPmod.

Can you upload an original untouched image sequence somewhere? 50 frames or so...

Fred.
100 frames of the source for this script are temporarily at: http://meloware.com/source/

Why Blender 3D? Blender is my Open Source "Swiss Army Knife" for graphics. I have been using it for 20 years and still haven't explored everything possible with it.
Blender has great potential in movie film restoration. The Compositor feature has a large bag of tools. There is the usual contrast/brightness and scale, flip, etc., but the color correction, gamma adjustments, and center box imaging positioning (translate) are very powerful. Perhaps the most important feature is that any of the Compositor settings may be animated. This means any adjustments in frame position, gamma, white/black levels and color corrections may be applied temporally, to compensate for momentary exposure or position. Professional film sources are well made for exposure, lighting, etc. Moment to moment adjustment for exposure and gamma are infrequent. Amateur films, however, usually have horrible lighting and can be greatly improved if you make the effort to make adjustment for each scene. Blender also features a tracker/stabilizer node. I have improved my work, even more, by using that feature in addition to AviSynth's stabilizer.

Blender is free and preparing for a major new release. I have included (in the source directory, link above) an example Blender file I used in one of my films. The source files are missing, but you can see the compositor setup and temporal edits I made.

AviSynth is amazing, and I am grateful for your creation of these scripts and the support I am getting. As I learn, I may find better alternatives to problems I now address in Blender. AviSynth appears to apply it's filters throughout a video according to the static script settings. I don't see any feature which allows temporal adjustment to change settings from moment to moment. You also need fast manual interaction to be able to edit these momentary overrides. Blender gives me the platform to make the final tweaks before rendering a final frame sequence.
Meloware is offline   Reply With Quote
Old 20th June 2019, 16:39   #1238  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Just a little question, is this script known to work ok (fast) where source are image files rather than video clip ?

EDIT: Perhaps try timing with same set up on video source, any difference ?
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 20th June 2019 at 16:43.
StainlessS is offline   Reply With Quote
Old 20th June 2019, 16:45   #1239  |  Link
Meloware
Registered User
 
Join Date: Dec 2018
Location: Massachusetts USA
Posts: 25
Quote:
Originally Posted by Groucho2004 View Post
Add these lines to the very top of your script:
Code:
SetMemoryMax(16000) #increase to 32000 or even 40000 if necessary.
threads = 16 #increase gradually up to 32 or 40. Any higher than that, the memory consumption will get close to your 64G (I think)
### Check speed, memory consumption and CPU usage with AVSMeter64 ###
Change the last line in your script ("Prefetch(4)") to this:
Code:
Prefetch(threads)
Also, put avstp.dll into your plugins64+ directory. It appears to be quite beneficial for this particular script.
Wow -what a change! I have never seen this CPU so active. The same script is now rendering at 2.5 frames per second. The script has threads set at 16, yet the CPU is at 96% utilization with all 64 threads fully active. Memory consumption is at 20% of the available 64GB. I don't believe I have any more computer available to improve this performance more.
How much of this change is due to the new avstp.dll ? I noticed the zip containing a 32 bit version, and I installed that as well, in the appropriate 32 bit plugin directory.
I haven't tried AVSMeter64, as yet, but CPUID is the source for my system performance data, and ffmpeg reports the 2.5 fps frame render rate in the CMD box.
You guys are going to get tired of me saying "Thank you"!
Meloware is offline   Reply With Quote
Old 20th June 2019, 16:58   #1240  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
I assume the breakthrough increase in speed was from the avstp.dll addition since it didn't appear that he was getting any MT even with the hard-wired threads(4) statement. I also assume that the SetMemoryMax increase had no impact on speed since, in my experience, memory allocation is entirely a setting for stability, not performance.

Feel free to correct me, if these assumptions are incorrect.

BTW, the next thing to try is to change the threads setting to find the "sweet spot." This can sometimes be counter-intuitive, given the strange way in which multi-threading works in various flavors of AVISynth. The only way I've been able to tune this is to run the script for 30-60 seconds, and then note where the fps results end up after that time (the initial fps can be quite misleading because of caching from a previous run). Then, I change threads up or down, and repeat.

I only have eight cores, and with my system, with most scripts (including this film restoration script), threads=8 is almost always really slow and unstable. Based on experience, I usually start my tuning with threads=4 (i.e., half of my cores), and work up or down from there. For many of my scripts, I get the best performance, with stability, with threads=5, but I've had to go as low as 3 and sometimes have gotten as high as 7, so I have learned that I must always do these brief tests.

Last edited by johnmeyer; 20th June 2019 at 17:00. Reason: re-format for clarity
johnmeyer is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 18:10.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.