View Single Post
Old 14th September 2018, 22:30   #13  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Hands-on tutorial (part 2/2)

While the optimizer is running we can run the visualization. Open another command line window in the optimizer's folder and run:

Code:
optimizer -mode evaluate
A window opens with a chart of the latest optimization run. I took a shot while the third run was under way:



The grey dots are all the results of the latest run. The red dots which are connected by a black line are the latest run's pareto front. The black larger dots, connected by a grey line, are the global pareto front, meaning the best results from all the runs. These descriptions are also visible in the info box in the bottom right corner. The info box also shows the latest run number and the number of iterations finished.

Over the chart window there's a long text, that's the name of the latest log file where these results are stored. The name is long because it contains all the relevant parameters of the metaheuristic algorithm. It also contains the script name we're optimizing and the timestamp when the optimization started. All this information could be useful later when you have hundreds of log files.

Notice how all the results are clustered on 10ms marks on the X axis, that's because the runtimes are rounded. The chart doesn't show all the results but instead focuses on the best ones. To be precise, the chart shows the best X% of the results. The focused percentage can be defined by a parameter, the default % is 20. This is implemented because the top results are usually the interesting ones. The default percentage can be changed with the argument -top.

Close the visualization window from the X and let's try an automatically updating chart. Start it by typing

Code:
optimizer -mode evaluate -autorefresh true
The chart opens like before but now you should see that the visuals are changing every few seconds. You can also see grey rectangles around some of the results. Those rectangles mean these results are among the last N results (default N is 10). If the optimizer starts a new run the autorefreshing chart starts showing it and the run number changes in the info box.

All right, that's enough about visualization for now. In the next phase we will look at the finished results. Go grab some coffee (or something else if you don't like coffee, like me) and come back in less than 8 minutes. Or even sooner if your machine is faster than mine.

...

Now that the optimizer has finished we can try to analyze the results. Start the evaluation once more (autorefresh not needed, there's nothing to refresh)

Code:
optimizer -mode evaluate
You can marvel the wonderful chart for a while if you like but we're going to focus on the command line window where we started the visualization from. It displays (among other things) the best result (again ranked by the first output's value) of every run. If there's a lot of variation in those results and their parameter values that means our search was not thorough enough. In that case you might get better results by increasing the iteration count. Here's what I got with 100 iterations:

Code:
Run 1 best: 4.940206 70 sigma=704 blockTemporal=5 blockSize=17 overlap=7
Run 2 best: 4.947239 50 sigma=551 blockTemporal=5 blockSize=13 overlap=6
Run 3 best: 4.935884 50 sigma=771 blockTemporal=5 blockSize=19 overlap=5
Run 4 best: 4.941647 90 sigma=607 blockTemporal=5 blockSize=23 overlap=10
Run 5 best: 4.928436 20 sigma=725 blockTemporal=3 blockSize=20 overlap=7
We can see that there's some variation in the SSIM values. And hey, 3 out of 5 are better than our goal, yippee! The parameter values are not that close to each other either. To see if more iterations would help I tried 200 iterations: (you don't need to run these)

Code:
Run 1 best: 4.94857 60 sigma=486 blockTemporal=5 blockSize=16 overlap=8
Run 2 best: 4.947047 60 sigma=543 blockTemporal=5 blockSize=16 overlap=8
Run 3 best: 4.931532 160 sigma=474 blockTemporal=5 blockSize=43 overlap=21
Run 4 best: 4.942368 30 sigma=464 blockTemporal=3 blockSize=12 overlap=6
Run 5 best: 4.944955 30 sigma=462 blockTemporal=4 blockSize=12 overlap=6
Hmm... not much of an improvement. Let's try 500 iterations:

Code:
Run 1 best: 4.9489594 60 sigma=480 blockTemporal=5 blockSize=14 overlap=7
Run 2 best: 4.948967 60 sigma=475 blockTemporal=5 blockSize=14 overlap=7
Run 3 best: 4.948965 60 sigma=485 blockTemporal=5 blockSize=14 overlap=7
Run 4 best: 4.948967 60 sigma=475 blockTemporal=5 blockSize=14 overlap=7
Run 5 best: 4.948962 60 sigma=477 blockTemporal=5 blockSize=14 overlap=7
Wow, look at that! All parameters but sigma are equal in all runs, and even sigmas are varying very little. Looks like we've found the optimal settings, or at least something very close to it. Just for kicks this is the result with 2000 iterations:

Code:
Run 1 best: 4.948967 60 sigma=479 blockTemporal=5 blockSize=14 overlap=7
Run 2 best: 4.948967 60 sigma=479 blockTemporal=5 blockSize=14 overlap=7
Run 3 best: 4.948967 60 sigma=479 blockTemporal=5 blockSize=14 overlap=7
Run 4 best: 4.948967 60 sigma=479 blockTemporal=5 blockSize=14 overlap=7
Run 5 best: 4.948967 60 sigma=479 blockTemporal=5 blockSize=14 overlap=7
The evaluation also shows the best and worst result of the series and the pareto front of the whole series (the global pareto front). These are the best results from all of the runs. The pareto front is sorted by the first output's value, best value first. Here's my pareto front from the 100 iterations series:

Code:
Pareto front:
  4.947239 50 sigma=551 blockTemporal=5 blockSize=13 overlap=6
  4.942367 40 sigma=566 blockTemporal=5 blockSize=16 overlap=6
  4.9421406 30 sigma=551 blockTemporal=4 blockSize=14 overlap=6
  4.93908 20 sigma=560 blockTemporal=3 blockSize=14 overlap=6
  4.917379 10 sigma=790 blockTemporal=3 blockSize=15 overlap=2
As you can see the first result has the best SSIM and the worst runtime, going down the list the SSIM gets worse and the runtime gets better. This is a pretty fast script to run, even the longest time was only 50 milliseconds so in this case we can forget about the runtime and choose the result with the best SSIM value. However, some other plugins can have wildly different runtimes depending on the used parameter values (MVTools is among those). In that case you should consider if the extra quality is worth the increased runtime.

One final step in this tutorial: let's create the scripts from the pareto front.

Code:
optimizer -mode evaluate -scripts true
After you've run that you should find Avisynth scripts with names like "denoise_PARETO_01_4.947239_50.avs". The name contains the script's name, the word PARETO, a number which stands for the order in the pareto front (01 -> best, 02 -> second best and so on) and the result values (SSIM and runtime). You can then run those scripts and compare the quality with your very own eyeballs.

So far we have always evaluated the latest optimization run. If you need to go back to earlier ones, give the name of the log file(s) with the -log parameter. You can use the "*" wildcard in the name, for example if you want to analyze all logs starting with "denoise" you could run

Code:
optimizer -mode evaluate -log "../test/flower/denoise*.log"
Note that quotes are needed when using the wildcards.

I will give you one more option to play with: the -vismode (short for "visualize mode") which can take the values none, single, series and seriespareto.
In the next episode we will take a closer look at the optimizer and how to customize the optimization process besides the iteration count.

Last edited by zorr; 14th September 2018 at 22:32. Reason: Corrected description of the chart
zorr is offline   Reply With Quote