View Single Post
Old 14th September 2018, 22:07   #12  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Hands-on tutorial (part 1/2)

[EDIT] Tutorial updated to match the features of the latest version of AvisynthOptimizer (no longer using VirtualDub to run scripts, better error handling, colored text)

It's finally time for you to get your greedy hands on AvisynthOptimizer! Download the zip here and unzip to a folder of your liking. You will also need my modded AvsTimer and the SSIM plugin with the SSIM_FRAME-function which returns the SSIM value to the script. I have a package which contains both of those here. Let's first see what is inside the AvisynthOptimizer package:

Code:
lib (folder)
AviSynthOptimizer.jar	
avsr.exe
avsr64.exe
optimizer.bat
optimizer.ini
versionHistory.txt
The optimizer is written in Java so you need to have Java Runtime. If it's not installed yet, go ahead and grab it from here.

AvisynthOptimizer.jar contains the main code. There are some external jar dependencies in the lib-folder.

avsr.exe and avsr64.exe this is Groucho2004's avsr utility which is used to run the scripts (thanks Groucho2004!)

optimizer.bat is a batch file you can use to run the optimizer.

optimizer.ini contains some configuration settings. In it's initial state it looks like this:

Code:
architecture=
log=
The first line tells the default Avisynth architecture (x86 or x64) used to run the scripts. The optimizer will ask you which one to use on the first time it's started (unless you're using 32bit Windows, in which case it doesn't need to ask) and write the value here. You can also override the default value using the -arch parameter.

The second line "log=" doesn't have a value either. The optimizer will update here the latest log file it has written to. It's useful when evaluating logs (we don't have to specify the log file, it defaults to the file found in this .ini file).

versionHistory.txt what could this be? yes, it's the version history.

Let's move on to our first optimizing task. When I was trying to come up what a suitable first demonstration I found this thread where Fizick was comparing the quality of denoisers using SSIM. One of the first filters he tried was FFT3DFilter. The script is short and simple and shouldn't be too difficult to optimize. Let's find out if we can beat Fizick's best settings!

The challenge introduced in that thread is the following: introduce noise to a video and then try to remove it with the denoiser, finally compare the denoised frames to the original frames with SSIM. The goal is to get as good similarity as possible. The video used can be downloaded here. It's in raw 4:2:0 YUV format and can we read with RawSource.

Let's first make a script that has the SSIM comparison and just for a good measure the timing measurement as well. If I was doing a serious optimization I would measure more than 5 frames but let's keep the runtime fast in this example. Note that we're adding the noise using a constant seed value so that the noise is exactly the same every time we run the script, making sure that there isn't an unknown variable messing with our quality measurements. The script is using the best FFT3DFilter settings found in the thread. Change the Rawsource path and resultFile path if needed. The example script has an absolute path for the source file but it's not necessary.

Code:
TEST_FRAMES = 5			# how many frames are tested
MIDDLE_FRAME = 50		# middle frame number

RawSource("D:\optimizer\test\flower\flower_cif.yuv", width=352, height=288, pixel_type="I420")
source=ColorYUV(levels="PC->TV")
noisy=source.AddGrain(25, 0, 0, seed=1)

denoised=noisy.FFT3DFilter(sigma=4, bt=4, bw=16, bh=16, ow=8, oh=8)		# best settings by Fizick

# cut out the part used in quality / speed evaluation
source = source.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
denoised = denoised.Trim(MIDDLE_FRAME - TEST_FRAMES/2 + (TEST_FRAMES%2==0?1:0), MIDDLE_FRAME + TEST_FRAMES/2)
last = denoised

global total = 0.0
global ssim_total = 0.0
FrameEvaluate(last, """
	global ssim = SSIM_FRAME(source, denoised)
	global ssim = (ssim == 1.0 ? 0.0 : ssim)
	global ssim_total = ssim_total + ssim	
""")	

# measure runtime, plugin writes the value to global avstimer variable
global avstimer = 0.0
AvsTimer(frames=1, type=0, total=false, name="Optimizer")

# per frame logging (ssim, time)
delimiter = "; "
resultFile = "perFrameResults.txt"	# output out1="ssim: MAX(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"
WriteFile(resultFile, "current_frame", "delimiter", "ssim", "delimiter", "avstimer")

# write "stop" at the last frame to tell the optimizer that the script has finished
frame_count = FrameCount()
WriteFileIf(resultFile, "current_frame == frame_count-1", """ "stop " """, "ssim_total", append=true)

return last
Save the script as "denoise.avs" and run it in VirtualDub (or whatever you normally use to run the Avisynth scripts) by double-clicking the file. In VirtualDub press play once and then look for the perFrameResults.txt file. It should contain something like this:

Code:
0; 0.987766; 12.808190
1; 0.987759; 6.340095
2; 0.987965; 6.358214
3; 0.987979; 5.819328
4; 0.987961; 5.781337
stop 4.939430
The quality values (0.987...) should be the same as above, your runtimes will be faster or slower than mine. The last line with the sum of SSIM values should be the same as well.

Ok, so 4.939430 is the result we're trying to beat. Now let's change the script a bit to make it ready for the optimizer.

Replace this part

Code:
denoised=noisy.FFT3DFilter(sigma=4, bt=4, bw=16, bh=16, ow=8, oh=8)		# best settings by Fizick
with this:

Code:
sigma = 400/100.0 	# optimize sigma = _n_/100.0 | 100..800 | sigma
bt = 4			# optimize bt = _n_ | -1..5 | blockTemporal
blockSize = 32		# optimize blockSize = _n_ | 2..64 | blockSize
overlap = 16		# optimize overlap = _n_ | 0..32 | overlap
denoised=noisy.FFT3DFilter(sigma=sigma, bt=bt, bw=blockSize, bh=blockSize, ow=overlap, oh=overlap)
When you start optimizing you have to guess what the suitable ranges are for each parameter. Since we already know at least one very good set of parameters we could set the range around those values. But let's not make it too easy for the optimizer and instead give it relatively large ranges. Besides, maybe the best result isn't near the values Fizick found, you never know.

We're going to try values 1.0 - 8.0 for the sigma, values from -1 to 5 for the bt (the full range allowed), values 2 - 64 for the bw and bh and 0 - 32 for the ow and oh. The last two are the overlap in x and y direction, they can use the same value because we have no reason to believe they should be different (at least in this case where the noise is uniform in all directions). The same applies for the block's size. In general it's a good idea to try to keep the number of parameters to optimize small, it makes the optimization job easier.

Now we can start the optimizer. Open a command line window in the directory you installed it to and write

Code:
optimizer <path_to_your_script>
The only thing you need to give is the path to the script you want to optimize. Mine was "../test/flower/denoise.avs".

Now since this is the first time you're running the optimizer it will ask the preferred Avisynth architecture:

Code:
Which Avisynth architecture are you (mostly) using?
NOTE: this default setting can be overridden with -arch argument
  1 - 32bit (x86)
  2 - 64bit (x64)
Just answer with 1 or 2 <ENTER>. The optimizer should start running and displaying the tested parameter combinations. However, it soon stops and displays an error message:

Code:
Error in script execution:

FFT3DFilter: Must not be 2*ow > bw
<script path and line number>
If you look at the last line you can see that the script is located at <optimizer_install_path>/work. That's the work folder where the optimizer is creating the scripts into. If there's an error the latest script will still be there so you can try to debug it. Also if you abort the optimization (with CTRL+C and answering Y when asked if you want to terminate the batch job) the optimizer doesn't have a chance to clean up properly so there may be some leftover script files in the work folder. It's safe to delete all of them if you're not currently running an optimization.

Let's figure out what the error message is trying to say. Looks like there is a dependency between the parameters ow and bw. We should add those to our script and try again. This dependency is almost the same we already saw in the MVTools script, namely that overlap cannot be larger than half the blockSize. The dependencies are thus:

Code:
blockSize = 32		# optimize blockSize = _n_ | 2..64 ; min:overlap 2 * | blockSize
overlap = 16		# optimize overlap = _n_ | 0..32 ; max:blockSize 2 / | overlap
Remember that we should define the dependency for both parameters involved in order not to introduce bias. Bias is bad! After you've replaced those lines run the optimizer again. This time it should be happy and keep on going.

Code:
Mutating 2 params by 28,5 %
RESOLVED: blockSize 5 -> 14
RESOLVED: blockSize 10 -> 36
RESOLVED: overlap 10 -> 6
RESOLVED: overlap 4 -> 3
* 105 / 2000 : 4.946868 50ms sigma=514 blockTemporal=5 blockSize=11 overlap=5
  106 / 2000 : 4.9306154 90ms sigma=717 blockTemporal=5 blockSize=26 overlap=13
  107 / 2000 : 4.911076 30ms sigma=726 blockTemporal=4 blockSize=5 overlap=2
  108 / 2000 : 4.932592 60ms sigma=732 blockTemporal=5 blockSize=14 overlap=7
+ 109 / 2000 : 4.8711557 10ms sigma=752 blockTemporal=4 blockSize=16 overlap=0
  110 / 2000 : 4.933645 70ms sigma=625 blockTemporal=5 blockSize=36 overlap=18
  111 / 2000 : 4.937391 60ms sigma=670 blockTemporal=5 blockSize=12 overlap=6
  112 / 2000 : 4.918705 30ms sigma=752 blockTemporal=4 blockSize=7 overlap=3
Parameter sensitivity estimation with 256 result combinations
  -> sigma 1,573 blockTemporal 1,178 blockSize 0,600 overlap 0,720
This is a short sample from my run. Every once in a while the optimizer tells what mutation settings it is currently using. The mutations are large in the beginning and gradually become smaller towards the end. This is not particularly relevant information, just a small sanity check.

Next we have four RESOLVED lines, they tell us how the conflicts were resolved in the current generation. If you don't see any resolve lines that just means that there weren't any conflicts. The important thing here is that both blockSize and overlap get their values changed. If you have a parameter with dependencies defined but never see it on the resolved line, then something could be wrong about the definitions.

Then we have a list of results. The first number is the current iteration, always increasing by one with every result. The second number is the total number of iterations in this run. The default is 2000. The next two numbers are the SSIM value and runtime which the script calculated. The rest of the line spells out the parameters used with this result. Some of the result lines have different colors and symbols (+, *) in front of them. If you're lucky you might even see the "e" symbol. The * symbol (red line) means this is the best result found so far. Now this needs a bit more explanation... didn't we already conclude that there is not one best result but a pareto front? Yes, but here the best result is determined by sorting the results by the *first* value output from the script (in this case the SSIM value). So whenever we find the best SSIM so far the * is displayed in front of the result. The + symbol (yellow line) signifies that we have found a new pareto front member. And if you see "e", it means this result is exactly as good as some other result already in the pareto front ("e" is short for "equal"). The optimizer never tries the same parameter combination twice so this should be pretty rare, but can happen because sometimes certain parameter don't have much (or any) effect on the result. Whenever you see * or + you know the algorithm is making progress.

The last two lines say some gibberish about parameter sensitivity. This is my invention where the algorithm is trying to determine (based on recent results) how sensitive the parameters are. A sensitive parameter is one that causes a large change in the result when its value changes. The sensitivity value shown is large for sensitive parameters and small for non-sensitive parameters. The reason for this sensitivity business is that the mutations we're making are scaled by the sensitivity in order to avoid too large or too small changes for the parameters. In my testing the optimizer gives better results with the sensitivity estimation than without it so it defaults to being enabled.

2000 iterations is a bit long for this optimization task, so let's stop the optimizer (CTRL+C, Y) and make it run a much lighter task. This time we give it the number of iterations as an argument:

Code:
optimizer <path_to_your_script> -iters 100
Now the optimizer will finish much faster, on my machine it only takes about 80 seconds. But wait a minute... it didn't finish after 100 iterations but started another run! Why is it doing that? Here's the deal: randomness is a large factor in the metaheuristic algorithms. This means that the results of an optimization run will be different every time, and the differences can be large. So if you only run the optimization once and have bad luck you get bad results. This is why you should try at least a couple of runs. The default is 5 runs. Let's wait until the optimizer has finished them all. Except that we still have something fun to try...

Last edited by zorr; 9th December 2018 at 23:41. Reason: Updated download link to latest version 0.9.16-beta
zorr is offline   Reply With Quote