Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
9th October 2018, 23:47 | #61 | Link | |||||||||||
Registered User
Join Date: Mar 2018
Posts: 447
|
Thanks, much appreciated.
That may not be the fault of Avstimer. The optimizer sets this value as the timing result when the SSIM is invalid (that is, zero). It's done that way in order to keep the invalid results out of pareto front (an invalid result might be faster than any other result and stay in the pareto front due to being the fastest). You could check what Avstimer really returns by running the script manually (in VirtualDub or Avsr for example) and see what's inside the perFrameResults.txt. Paste it here as well, maybe there are more clues as to what is wrong. Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Code:
super_pel = 2 # optimize super_pel = _n_ | 2,4 | super_pel super_sharp = 2 # optimize super_sharp = _n_ | 0..2 | super_sharp super_rfilter = 4 # optimize super_rfilter = _n_ | 0..4 | super_rfilter Last edited by zorr; 9th October 2018 at 23:58. Reason: Calculated combinations was 900 times too small |
|||||||||||
10th October 2018, 00:06 | #62 | Link | |
Registered User
Join Date: Mar 2018
Posts: 447
|
Quote:
|
|
10th October 2018, 00:14 | #63 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
Edit: Checked the time stamp on the first one you posted - It's Kassandro's DLL from 2005.
__________________
Groucho's Avisynth Stuff Last edited by Groucho2004; 10th October 2018 at 00:20. |
|
10th October 2018, 00:33 | #65 | Link |
Registered User
Join Date: Mar 2018
Posts: 447
|
Plugins package with SSIM and AvsTimer has been updated, hopefully with a more functional 32bit AvsTimer.
|
10th October 2018, 00:47 | #66 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
__________________
Groucho's Avisynth Stuff |
|
10th October 2018, 02:43 | #67 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Quote:
Unfortunately, not everyone is willing to go to the bother of compiling Ye Olde stuff, even if source does exist. Above link has been there for several years I think, not altogether a bad idea to have access to such when required. (EDIT: not needed in this case when there is someone as affable as Ye Olde G2K4 )
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 10th October 2018 at 02:55. |
|
10th October 2018, 08:06 | #68 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
__________________
Groucho's Avisynth Stuff |
|
10th October 2018, 18:44 | #69 | Link | |
Registered User
Join Date: Sep 2010
Location: Russia
Posts: 85
|
The new avstimer reports time correctly, thanks.
Some other ideas to add, previously it was mentioned that some parameter combinations might have no effect on time or SSIM, but what if we knew for sure what those combinations are, it would be nice to be able to mark them as such so the optimizer could skip them. For example, in MSuper the sharp parameter is only for pel>1, it won't raise an error otherwise but it'll be a waste of time. Regarding the ConvertToYV24, can you give an example what kind of parameter combinations would be unsuitable for YV12 so this becomes necessary? It slows down the processing considerably, so I'd like to get rid of that. Another thing, the badRange parameter of MAnalyze says that we need to use positive values for UMH search and negative for exhaustive. Unfortunately it does not disclose why, however it doesn't raise an error anyway. But the FPS for negative values is almost a half of what's achieved for positive ones (when you set badSAD to 0 so the wide search is always invoked, not a real-life scenario, but might happen a lot during the search). In SVP a similar parameter is described as using an adaptive radius when negative to save time, claiming it takes about 2/3 of time for a similar result, but it seems we have the opposite here. Anyway, how would we describe it in the settings to use negative values when searchAlgo is 3? Finally, I wonder about the temporal parameter. The readme says it's incompatible with setMTmode, however the new mvtools have the MT parameter inbuilt and on by default, do we know if it should be disabled for temporal? Again, it doesn't raise errors, the output looks differently but then it also does look differently when disabling MT for temporal=false as well. Really, the readme should be updated there. To comment on your post about the runs vs iters, you said, Quote:
If I'm correct about that all we need to figure out is how to scale population count over the increasing search space. What configuration is more likely to try the largest subset of the search parameters, the high-iter low-pop or the vice-versa? Last edited by Seedmanc; 10th October 2018 at 21:30. |
|
11th October 2018, 00:24 | #70 | Link | ||||||||
Registered User
Join Date: Mar 2018
Posts: 447
|
Quote:
Quote:
Code:
super_pel = 2 # optimize super_pel = _n_ | 1,2,4 | super_pel super_sharp = 2 # optimize super_sharp = _n_ | 0..2 ; max:super_pel 1 == 0 2 ? | super_sharp Quote:
Code:
overlap=0 # optimize overlap=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / | overlap overlapv=0 # optimize overlapv=_n_ | 0,4,8,12,16,20,24,28,32 ; max:blockSize 2 / | overlapv Quote:
Quote:
Quote:
Quote:
There is another reason to do multiple runs. The beginning of the search usually "locks" the search into a certain corner of the search space and it may never get out of that within the iteration count. So it could be that you get significantly better result in one out of say, 10 runs. If you only ever do 3 runs maybe you'll never get to that lucky corner. Here's a recent example: Code:
Run 1 best: 9.790725 2130 rmgrain=12 super_rfilter=1 blockSize=8 searchAlgo=3 searchRange=1 searchRangeFinest=7 lambda=2213 LSAD=2744 plevel=2 overlap=0 globalMotion=true badSAD=9559 badRange=34 meander=false temporal=false trymany=false dct=1 maskScale=2 Run 2 best: 9.787563 1210 rmgrain=19 super_rfilter=1 blockSize=8 searchAlgo=1 searchRange=1 searchRangeFinest=3 lambda=3931 LSAD=19316 plevel=2 overlap=0 globalMotion=false badSAD=2770 badRange=13 meander=true temporal=true trymany=false dct=1 maskScale=2 Run 3 best: 9.790011 1510 rmgrain=19 super_rfilter=1 blockSize=8 searchAlgo=3 searchRange=1 searchRangeFinest=4 lambda=2629 LSAD=1684 plevel=2 overlap=0 globalMotion=true badSAD=9831 badRange=10 meander=true temporal=false trymany=false dct=1 maskScale=2 Run 4 best: 9.789404 1190 rmgrain=19 super_rfilter=0 blockSize=8 searchAlgo=1 searchRange=1 searchRangeFinest=2 lambda=2142 LSAD=2679 plevel=2 overlap=0 globalMotion=true badSAD=9558 badRange=45 meander=true temporal=false trymany=false dct=1 maskScale=2 Run 5 best: 9.788453 1220 rmgrain=8 super_rfilter=0 blockSize=8 searchAlgo=1 searchRange=1 searchRangeFinest=3 lambda=3432 LSAD=6410 plevel=2 overlap=0 globalMotion=true badSAD=4531 badRange=4 meander=true temporal=false trymany=false dct=1 maskScale=1 Quote:
Strictly speaking there is no difference, the number of iterations defines how large the searched subset is (as the optimizer never tries duplicates within one run). But I guess you're asking how to get the widest possible subset. For that question the high population should do better in terms of how wide the search is but it's going to do less mutations of the best result and therefore might end up with a worse result than a smaller population. The "mutation" algorithm with population 1 is the most narrow search possible, it simply keeps the best result and mutates it until one of the mutations is better. If you just want to make the search wider, you can do that also by cranking up the mutation amount and count. Last edited by zorr; 11th October 2018 at 00:42. Reason: Added suggestion to use mutation count / amount |
||||||||
12th October 2018, 18:25 | #71 | Link | |
Registered User
Join Date: Sep 2010
Location: Russia
Posts: 85
|
Two problems, one is that after I modified the script to disable YV24 conversion, limit the sharp parameter and introduce the negative badRange (didn't work with -50..50 values so I added a boolean flag according to which I do or do not multiply the range by -1), the script started crashing, complaining that "MflowFps can't work in reentrant multithreading" even though I didn't touch anything MT-related. While I do have MT versions of Avisynth 2.6 and Avisynth+ installed I do not call setmtmode or prefetch in either but it crashes in both randomly, I'm lucky if I manage to get one run finished.
I used to get this error before AvsOptimizer sometimes too, usually when switching between multiple heavy scripts in Avsp editor, but it was a rare occurrence. Before I edited the script I managed to run it overnight totalling in over a thousand iterations without a problem. If it didn't work right away I'd consider that launching two upsampling operations in succession like we do here might've been the culprit. Here's my script as of now: Quote:
Also, for some reason sensitivity estimation is always marked as N/A in the log, even if I try to pass -sensitivity true. It would be interesting to see which parameters are more important. |
|
12th October 2018, 21:14 | #72 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
__________________
Groucho's Avisynth Stuff |
|
12th October 2018, 23:36 | #73 | Link | ||
Registered User
Join Date: Mar 2018
Posts: 447
|
Quote:
Code:
SetMemoryMax(2048) I like the way you implemented the positive / negative badRange. It's pretty much the way I would have done it, separating the algorithm into its own parameter. Quote:
The mutation algo is not quite as fully featured as the others, it's missing the sensitivity estimation at the moment. I will add it in the next version. It's probably not going to happen during this weekend though, I'm moving tomorrow and about to start renovating the new house... Last edited by zorr; 12th October 2018 at 23:55. Reason: Added info about testing the script |
||
13th October 2018, 00:12 | #74 | Link |
Registered User
Join Date: Sep 2010
Location: Russia
Posts: 85
|
Groucho2004, here are the logs: https://pastebin.com/trPLFs2w (primary setup with avs2.6), https://pastebin.com/TjdMquUr (secondary setup with avs+). For some reason it complains about the absence of fft3w lib in the latter, even though mvtools dct modes work fine.
zorr, thanks for the memory suggestion and the time fix, I'll try to run it overnight now with those. I wonder if it would be possible to use the 3Gb or 4Gb limits reliably since I'm on 64bit OS. SEt says it requires the patching of the runner app (avsr in our case) for support, I assume it applies regardless of MT usage, does avsr have that? |
13th October 2018, 02:12 | #75 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
As for fft3w - Just ignore it for now.
__________________
Groucho's Avisynth Stuff |
|
13th October 2018, 02:56 | #76 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
W10
Code:
[CPP 2.5 Plugins (32 Bit)] C:\Program Files (x86)\AviSynth+\plugins+\avstimer.dll [2018-10-07] # 2.5 Plugin in Plugins+ folder C:\Program Files (x86)\AviSynth+\plugins+\ffms2.dll [2013-05-21] # 2.5 Plugin in Plugins+ folder [CPP 2.6 Plugins (32 Bit)] C:\Program Files (x86)\AviSynth+\plugins\avstimer.dll [2018-09-11] # see above. (earlier plug is v2.6, later 2.5 ?????? ) C:\Program Files (x86)\AviSynth+\plugins+\mvtools2.dll [2.5.11.22] # Plugins/+ mixup with below C:\Program Files (x86)\AviSynth+\plugins\mvtools27.dll [2.7.31.0] # ditto EDIT: The 'hazy' part is standard v2.6 plug with standard VERSION 6 header, versus avs+ VERSION 6 Header, ie should avs+ VERSION 6 header but without avs+ colorspace or functionality be in + or standard plugins (?).
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 13th October 2018 at 08:23. |
13th October 2018, 03:21 | #77 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
Besides, since there is a 64 bit version of the Optimizer, why not use that? Lastly, why would you use SEt's ancient Avisynth MT? AVS+ is quite stable, has a 64 bit version, uses less memory and, from my experience, has better MT support.
__________________
Groucho's Avisynth Stuff Last edited by Groucho2004; 13th October 2018 at 03:25. |
|
13th October 2018, 12:18 | #78 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
There's no reason to use any other header for 2.6 plugs than the one that comes with AVS+ which is fully backward compatible with classic AVS.
__________________
Groucho's Avisynth Stuff Last edited by Groucho2004; 13th October 2018 at 12:21. |
|
14th October 2018, 12:53 | #79 | Link |
Registered User
Join Date: Sep 2010
Location: Russia
Posts: 85
|
So, after much suffering (because nothing helped against the crashes, not even running it on 64bit avs+) I figured out to take a look at the intermediate avs scripts generated by the optimizer to find what has brought it to its knees. https://pastebin.com/bgUELWu6 this one crashes 100% of the time, can you tell what the problem is?
I'll save you time, it's the combination of pel 4, blocksize 8 with overlap 4, divide > 0 with large search radius and (surprise!) removal of ConvertToYV24. What does it have to do with MT? Well I don't know, ask the mvtools maker about their error reporting style. Further testing revealed that for pel 4 it is enough to have search radius of 4 to cause error, with pel 2 it takes around 12 and I couldn't reproduce it for pel 1. While I admit that having a search radius larger than block size seems strange it doesn't cause an error for YV24 or divide 0 / pel 1. What's more weird is that unlike the chroma subsampling violation it does not necessarily raise an error right away, sometimes it happens in the middle or at the end of the script, sometimes it's not the MT error but some random access violation. Moreso, setting chroma to false doesn't help (but converting to YV24 still does even then). Nothing in the readme has prepared me for this. Worst thing I don't even know how to report it, the thread has been abandoned for months. I guess more filters/minmaxes are in order, but with the current notation it's hard to figure out how to write them. Set divide to 0 when blocksize is 8 and overlap 4 and pel > 1. In a way, Optimizer can be used as an automated plugin testing tool, since it tries out so many parameter combinations and reveals all kinds of bugs and readme inconsistencies. Also, zorr, you might want to link you large explanative posts from this thread in the first post, now that the discussion took off it'll be more difficult to find them later. Last edited by Seedmanc; 14th October 2018 at 13:27. |
14th October 2018, 16:36 | #80 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Quote:
Seems that the [EDIT: forum] spell checker works again, been missing for some time (or maybe its because I'm on W7_64 at the moment, instead of XP32/64). [EDIT: Or maybe spell check is via current version of Firefox for W7, and no longer supported for last for XP FireFox v52.xx ESR]
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 15th October 2018 at 00:45. |
|
|
|