View Single Post
Old 9th September 2018, 01:22   #1  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 447
Introducing Zopti (ex AvisynthOptimizer)

Last year I dug up some old VHS videos and decided to finally digitize them. The tapes are old and I wanted to preserve them in the best quality I could afford. I soon found AviSynth and started to investigate how to do restoration and deinterlacing (for playback on computer).

This lead me to QTGMC-plugin and I was impressed by its capabilities but I did find some strange artifacts when I used the best quality settings. These artifacts were not present at lower quality settings. Being a perfectionist I naturally searched manually which of the 92 parameters were responsible for the artifacts and found them (more about that later).

There is quite a bit of noise in the old VHS tapes so naturally I investigated noise removal plugins. The temporal denoising made possible by MVTools is very important and I wanted to find out which settings are best for my VHS tapes. Reading these and other video restoration forums I found that there really isn't any one set of settings that would work for every type of video. So finding good settings would involve a lot of manual testing. Using MVTools to do motion compensation involves about 60 parameters so there's a lot of things to adjust... this task seemed hopeless.

So I began to wonder if there is anything that could automate this kind of process. Turns out that there is, and I made a tool that can help doing it. And now I'm about to share it with you.

An AviSynth script is first "augmented" by adding instructions on which parameters should be optimized. The instructions are written inside comments so the script will continue to work normally. The script also needs to measure the quality of the processed video using current parameter values or measure the running time (or preferably both the quality and runtime). The script needs to write these results to a specific file.

Now you might be wondering how on earth can the script measure the quality. I can only think of one way: to compare the frames to reference frames and measure the similarity. The closer the similary value, the better the quality. You could measure similarity in a very simplistic way using LumaDifference and ChromaUDifference+ChromaVDifference, but the best similary metric AviSynth currently has is SSIM. SSIM is not the best similarity metric anymore but it's still widely used in the image processing community. The SSIM plugin needs to have a function which returns the similarity value to the script. The only version that currently does that (as far as I know) is v0.25.1 by mitsubishi.

Ok, but where do we get the reference frames? In case of MVTools we can use the original frames as reference frames and the script will try to reconstruct them using motion compensation (but it's not allowed to use the reference frame in the reconstruction). We can do something similar if we want to use MVTools to double the framerate: we first create the double-rate video, then remove the original frames from it, then double the framerate again and finally compare these to the original frames. This idea is not limited to MVTools, you could for example do color grading using some other software and then try to recreate the same result using AviSynth. I'm sure the smart people here will find use cases I couldn't even dream about.

Most people don't only care about the quality, the script's run time is also important. I found a couple of different plugins to measure the runtime, but only one of them (AvsTimer) fit the bill and even then I had to make some modifications to it.

So if the purpose is to find the best settings, what do we consider the "best" when both quality and time are involved? For example we can have settings A with quality 99 (larger is better), time 2300ms (larger is worse) and settings B with quality 95 and time 200ms. Which one of these is better? We can use the concept of pareto domination to answer this question. When one solution is at least as good as the other solution in every objective (in this case the objectives are quality and speed) and better in at least one objective, it dominates the other solution. In this example, neither dominates the other. But if we have settings C with quality 99 and time 200ms it dominates both A and B. In the end we want to know all the nondominated solutions, which is called the pareto front. So there's going to be a list of parameters with increasing quality and runtime. You can have more than two objectives if you want, the same pareto concept works. There's a good and free ebook called "Essentials of Metaheuristics" which describes the pareto concept and much more.

AvisynthOptimizer reads the augmented script, verifies it and starts running a metaheuristic optimization algorithm. There are currently three algorithm choices: NSGA-II, SPEA2 and mutation. The first two are some of the best metaheuristic algorithms available and also described in the ebook mentioned above. The mutation is my own simplistic algorithm (but it can be useful if you're in a hurry since it's the most efficient). All the metaheuristic algorithms are working in a similar manner generating different solutions and testing them. The optimizer does these tests by creating AviSynth scripts with certain parameter values and running them. The script then writes quality/time results into a file which the optimizer then reads. The metaheuristic then decides which parameters to try next based on the results. This continues until some end criteria is met.

There are three different ending criterias: number of iterations, time limit and "dynamic". Number of iterations is just that, the algorithm runs the script specific number of times. Setting a time limit can be pretty useful if you know how much time you can use for the optimization. You could for example let it run overnight for 8 hours and see the results in the morning. Dynamic variation is stopping only when it doesn't make any progress anymore. Making progress is defined by "no more pareto front members in last x iterations". This can be useful if you want to find the best possible results regardless of how long it takes.

During the optimization all tested parameters and their results are written to a log file. This log file can be "evaluated" during and after the optimization process. The evaluation basically means finding the pareto front and showing it. You can also create AviSynth scripts from the pareto front in order to test them yourself. It's also possible to visualize the results of the log file in a two dimensional scatter chart. This chart highlights the pareto front and shows all the other results too. The chart can also be "autorefreshing": it loads the log file every few seconds and updates the visuals, which is a fun way to track how the optimization is progressing. Here's a gif what it looks like (obviously sped up):



The visualization has a few other bells and whistles but one I'd like to highlight here is the group by functionality: you can group the results by certain parameter's values and show a separate pareto front for each value. I think a picture is in order here, this what grouping by MVTools' blocksize looks like:



Measuring the script's runtime is not very accurate, ie it has some variation. All the other processes running at the same time are using CPU cycles and messing with the cache so you should try to minimize other activity on the computer. In order to get more accurate results you can run a validation on the finished results log file. In validation the idea is to run the pareto front results multiple times and calculate the average, median, minimum or maximum of these multiple measurements (you can decide which one(s)). After the pareto front is measured validation calculates the "secondary pareto front" by removing the results in the original pareto front and finding a pareto front of the remaining results. The verification is run on the secondary pareto front also because it's possible that due to inaccurate runtimes the real pareto front will contain results from the secondary pareto front. If the secondary pareto front did contain real pareto front members the validation takes the third pareto front and validates it as well. And so on until the current pareto front didn't contain any new pareto front members.

So how good is the optimizer? Let's take a look at one example. A while back there was a thread about best motion interpolation filters. There's a test video with a girl waving her hand. The best options currently are John Meyer's jm_fps script and FrameRateConverter. Here's a comparison gif with those two and AvisynthOptimizer. FramerateConverter was run with preset="slow".



Now obviously I'm showing a bit of a cherry-picked example here. The optimizer was instructed to search the best parameters for this short 10 frame sequence. I have run most of my optimization runs using only 10 frames because otherwise the optimization takes too long. Ideally the optimizer would automatically select the frames from a longer video, I have some ideas on how to implement that but as of now the user has to make the selection. Also this part of the video is not the most challenging part, I decided to try something relatively easy first. After all the optimizer cannot do miracles (that feature is not finished yet ).

At this point I envision that AvisynthOptimizer would be an useful tool for plugin authors so they can test plugin parameters and try to search for the optimal ones. At some later point AvisynthOptimizer could be useful for normal users, when combined with a script with limited search space so that the search will not take excessively long time.

All right, that's all for now. I will add more detailed explanations and the download links tomorrow. I haven't actually finished the documentation yet, but still wanted to get this thing out there before my vacation is over.

[EDIT] Here are the download links:
Download the zip (version 1.2.3) here and unzip to a folder of your liking. You will also need my modded AvsTimer and the SSIM plugin with the SSIM_FRAME-function which returns the SSIM value to the script. I have a package which contains both of those here.

More documentation can be found from these posts (if you're not interested in reading the whole thread):
Augmented Script (part 1/2)
Augmented Script (part 2/2)
Hands-on tutorial (part 1/2)
Hands-on tutorial (part 2/2)
Optimizer arguments

Last edited by zorr; 5th March 2022 at 21:24. Reason: Changed download link to latest version 1.2.3
zorr is offline   Reply With Quote