View Single Post
Old 10th September 2018, 01:20   #8  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Augmented Script (part 1/2)

Let's take a look at what an "augmented" script looks like. This is a complete script that contains everything AvisynthOptimizer needs. The script is using MFlowInter to reconstruct every frame of the video using the neighbour frames and then compares them to the original frames.

Code:
TEST_FRAMES = 10		# how many frames are tested
MIDDLE_FRAME = 50		# middle frame number

AVISource("d:\process2\1 deinterlaced.avi")
orig = last

# you could add preprocessing here to help MSuper - not used here
searchClip = orig

super_pel = 4					# optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2					# optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 2				# optimize super_rfilter = _n_ | 0..4 | super_rfilter

super_search = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, searchClip)		
super_render = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, last, levels=1)	

blockSize = 8					# optimize blockSize = _n_ | 6,8,12,16,24,32 ; min:divide 0 > 8 2 ? ; filter:overlap overlapv max 2 * x <= | blockSize
searchAlgo = 5					# optimize searchAlgo = _n_ | 0..7 D | searchAlgo
searchRange = 2					# optimize searchRange = _n_ | 1..10 | searchRange
searchRangeFinest = 2				# optimize searchRangeFinest = _n_ | 1..10 | searchRangeFinest
lambda = 1000*(blockSize*blockSize)/(8*8)	# optimize lambda = _n_ | 0..20000 | lambda
lsad=1200					# optimize lsad=_n_ | 8..20000 | LSAD
pnew=0						# optimize pnew=_n_ | 0..256 | pnew
plevel=1					# optimize plevel=_n_ | 0..2 | plevel
overlap=2					# optimize overlap=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / ; filter:x divide 0 > 4 2 ? % 0 == | overlap
overlapv=2					# optimize overlapv=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / | overlapv
divide=0					# optimize divide=_n_ | 0..2 ; max:blockSize 8 >= 2 0 ? overlap 4 % 0 == 2 0 ? min | divide
globalMotion = true				# optimize globalMotion = _n_ | false,true | globalMotion
badSAD = 10000					# optimize badSAD = _n_ | 4..10000 | badSAD
badRange = 24					# optimize badRange = _n_ | 4..50 | badRange
meander = true					# optimize meander = _n_ | false,true | meander
temporal = false				# optimize temporal = _n_ | false,true | temporal
trymany = false					# optimize trymany = _n_ | false,true | trymany

# smallest delta is 1 but you can make the task more challenging by using larger delta (larger deltas often used in temporal denoising)
delta = 1

useChroma = true
bv = MAnalyse(super_search, isb = true, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, \
delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, divide=divide, badSAD=badSAD, \
 badrange=badRange, meander=meander, temporal=temporal, trymany=trymany)
fv = MAnalyse(super_search, isb = false, blksize=blockSize, search=searchAlgo, searchparam=searchRange, pelsearch=searchRangeFinest, chroma=useChroma, \
delta=delta, lambda=lambda, lsad=lsad, pnew=pnew, plevel=plevel, global=globalMotion, overlap=overlap, overlapv=overlapv, divide=divide, badSAD=badSAD, \
 badrange=badRange, meander=meander, temporal=temporal, trymany=trymany)

# NOTE: we disable scene change detection by setting thSCD1 very high
blockChangeThreshold = 10000 
maskScale = 70								# optimize maskScale = _n_ | 1..300 | maskScale
inter = last.MFlowInter(super_render, bv, fv, time=50, ml=maskScale, thSCD1=blockChangeThreshold, thSCD2=100, blend=false)

# SSIM needs YV12 colospace
inter_yv12 = inter.ConvertToYV12()
orig_yv12 = orig.ConvertToYV12()

# for comparison original must be forwarded one frame
orig_yv12 = trim(orig_yv12,1,0)

# cut out the part used in quality / speed evaluation
inter_yv12 = inter_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
orig_yv12 = orig_yv12.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = inter_yv12

# calculate SSIM value for each test frame
global total = 0.0
global ssim_total = 0.0
FrameEvaluate(last, """
	global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
	global ssim = (ssim == 1.0 ? 0.0 : ssim)
	global ssim_total = ssim_total + ssim	
""")	

# measure runtime, plugin writes the value to global avstimer variable
# NOTE: AvsTimer should be called before WriteFile
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
delimiter = "; "
resultFile = "D:\optimizer\perFrame.txt"	# output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="D:\optimizer\perFrame.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
frame_count = FrameCount()
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

# return original and reconstructed frame side by side for comparison
#return StackHorizontal(orig_yv12, inter_yv12)

# NOTE: must return last or FrameEvaluate will not run
return last
That's not the most simple example I could think of but it's a good example because it uses some of the more advanced features of the optimizer.

Code:
TEST_FRAMES = 10		# how many frames are tested
MIDDLE_FRAME = 50		# middle frame number
We begin by declaring a couple of variables which help choosing the part which gets optimized. TEST_FRAMES tells how many frames long the test sequence is. MIDDLE_FRAME tells the frame number in the middle of the test sequence. So you can find an interesting part of the video and make note of the frame number and then set it as the middle frame number. These are by no means necessary in the script, they're just there to help.

Code:
super_pel = 4					# optimize super_pel = _n_ | 2,4 | super_pel
super_sharp = 2					# optimize super_sharp = _n_ | 0..2 | super_sharp
super_rfilter = 2				# optimize super_rfilter = _n_ | 0..4 | super_rfilter

super_search = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, searchClip)		
super_render = MSuper(pel=super_pel, sharp=super_sharp, rfilter=super_rfilter, last, levels=1)
Skipping a couple of lines we find the first AvisynthOptimizer related code. Here we define a couple of variables (in practice constants) that are later used in MSuper calls. The comment lines which start with "# optimize" are read by AvisynthOptimizer. (NOTE: "#optimize" will not work). The comments consist of three sections separated by "|" character.

The first section (after the word optimize) tells which part of code needs to be manipulated by the optimizer. The part with "_n_" is replaced with different values, the rest is there just give enough context on where this "_n_" -part is located. Whitespace matters here, so "# optimize super_pel=_n_" would not work when the code says "super_pel = 4".

The second section tells the valid values the optimizer should try for this parameter. You can define it as a range, for example "1..5" which would mean values from 1 to 5. Or you can define the values as a list separated by commas, for example "1,2,3". Booleans are supported (usually given as "false,true"). Strings are supported as well. Floats however are not, so if the parameter is a floating point number define it something like this:

Code:
param = 100/1000.0 	# optimize param = _n_/1000.0 | 0..1000 | paramName
This would give you floating point values between 0.0 and 1.0 with 0.001 intervals. You should consider what to use as the divider. A large divider means larger search space which makes the search more difficult. Too small divider could mean you miss the optimal value. One good strategy could be using a coarse interval first and do another run with finer interval after you've figured out an approximate optimal value. Don't forget the ".0" in the divider, otherwise you get integer division and no floating point numbers!

The last section is the name of the parameter. I have named some of the parameters differently than their corresponding parameter name in MVTools just to make it a little easier to remember what they do. These names are used in the log files where tested parameter values are reported. Also you can refer to other parameters by using these names (more about that in a little while).

Code:
searchAlgo = 5		# optimize searchAlgo = _n_ | 0..7 D | searchAlgo
This is another basic variable / optimizer definition but it has this "D" after the range. The D is short for "discrete" and means the optimizer should not make any assumptions that values closer to each other are more similar. Usually this is the case, for example when changing search range from 8 to 9 we expect smaller change in the result than if we change it from 8 to 20. This particular parameter controls the search algorithm and we have no idea whether algorithms 1 and 2 are more similar than, say, algorithms 1 and 3.

Code:
overlap=2	# optimize overlap=_n_ | 0,2,4,6,8,10,12,14,16 ; max:blockSize 2 / ; filter:x divide 0 > 4 2 ? % 0 == | overlap
What on earth is this? Calm down, there is an explanation. So the middle part where valid values are described can itself be divided into sections separated by ";"-character. The first part is just a normal value list: "0,2,4,6,8,10,12,14,16". The second and third parts are there because of an important feature: conflict resolvation (not sure if that's a word). So MVTools is kinda picky on what kind of parameter values you give to it. If it doesn't like the combination it will give an error message instead of doing something useful. For example overlap cannot be larger than half the blocksize. The optimizer doesn't know this (unless we tell it) and would happily try thousands of combinations where overlap is larger than blocksize/2. One way out of this problem is to wrap the script in try..catch and write out zero as quality when this happens. That would work but the optimizer would waste a lot of time trying invalid parameter combinations.

The better solution is to tell the kind of dependencies the parameters have between them. In this case we define a max dependency: "max:blockSize 2 /". The part after "max:" is a function written in reverse polish notation. For all the non-reverse-polish people this means blockSize/2. Now the optimizer knows that the maximum value of overlap is blockSize/2. Here we are referring to another parameter "blockSize" by its name.

The final part and the most difficult one by far is the filter dependency. It's used when minimum and maximum dependencies are not enough. The idea of the filter is that the optimizer will do a test for the current parameter values during the runtime. The current value of the parameter is put inside the formula replacing the "x". Then the formula is evaluated and if it returns true the parameter value is accepted as valid. The formula in reverse polish form "x divide 0 > 4 2 ? % 0 ==", which in infix form is "x % ((divide > 0) ? 4 : 2) == 0". "divide" is not division operator but another parameter name (perhaps not the best name choice here). In plain english this states that if (divide > 0) then overlap should be divisible by 4, otherwise it should be divisible by 2.

Code:
blockSize = 8		# optimize blockSize = _n_ | 6,8,12,16,24,32 ; min:divide 0 > 8 2 ? ; filter:overlap overlapv max 2 * x <= | blockSize
I will highlight this line as well because here in the filter definition we have "max", which is the maximum function. The full list of supported operators is +, -, *, /, %, <, >, <=, >=, ==, !=, ?, min, max, and, or.

Whenever you define a dependency with another parameter, you should do it for both parameters involved. So for example because overlap has a max dependency on blockSize, the blockSize should have min or filter dependency on overlap. The reason for this is to avoid bias in the search. If there is a conflict between overlap and blockSize, the optimizer will try to resolve it by changing either overlap's value or blockSize's value (and it does this in a fair way so that both get changed as often). If the dependency is defined only in one of the parameters, it will always change that parameter's value (it's not smart enough to figure out the valid values for the other one). This would introduce bias in the search and possibly ruin the chances of finding the optimal results.

That's all there is to defining the parameters to optimize. The reverse polish notation is something I would like to change since it's not the most user friendly format. The infix parsers I looked at were unfortunately not able to deal with ternary operators. But if you just want to optimize MVTools the hard part is already done and you can reuse my parameter definitions.

Code:
# calculate SSIM value for each test frame
global total = 0.0
global ssim_total = 0.0
FrameEvaluate(last, """
	global ssim = SSIM_FRAME(orig_yv12, inter_yv12)
	global ssim = (ssim == 1.0 ? 0.0 : ssim)
	global ssim_total = ssim_total + ssim	
""")
This is the part where we calculate the quality using SSIM. The current frame's quality is stored at "ssim"-variable and the code also calculates the total into "ssim_total". Sometimes MVTools returns the original frame when called with bad parameter values and SSIM returns the maximum similarity 1.0. Those frames have to be rated zero instead.

Code:
# measure runtime, plugin writes the value to global avstimer variable
# NOTE: AvsTimer should be called before WriteFile
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")
This part measures the runtime of every frame. The AvsTimer plugin needed some modifications for this purpose. The original returned frames per second as integer value, I changed it to return the passed time in milliseconds. Moreover, the timer was reporting the results into Windows debug log only, my version writes the result into a global variable called avstimer (I should add a parameter where you can define the parameter name).

Part 2 below...

Last edited by zorr; 23rd November 2018 at 00:45. Reason: Added new operators !=, and, or
zorr is offline   Reply With Quote