Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 23rd May 2020, 19:51   #201  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Quote:
Originally Posted by pinterf View Post
How much memory is required for all those 24 threads? Mvtools working area can be quite memory hungry for example, multiply it by 24 (filters are mostly MT_MULTI_INSTANCE mode) + UHD source. What is avsmeter telling? If it is near 4GB then use SetMemoryMax with e.g. 8000 or use a value which is probably above the memory need by a safe margin.
Pinterf, I thank you It was the default cache, which was not enough.. I raised it to 10000 and CPU usage jumped substantially to 90-100% with much less GPU usage than earlier.

I owe you one good pörkölt
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 23rd May 2020, 19:54   #202  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Could you pls do a test: keep the original default 4GB memory and set the other cache mode, I wrote about above. What are your numbers now?
pinterf is offline   Reply With Quote
Old 23rd May 2020, 20:28   #203  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Quote:
Originally Posted by manolito View Post
This is a follow-up to this post:
https://forum.doom9.org/showthread.p...13#post1912913


and I do agree with FranceBB on his conclusion.

My issues occur under Win7-64 on a Core i5 CPU with 8GB of RAM.

In any case I feel that AVS+ 3.6 is not ready for prime time yet. Please implement a more thorough quality control before publishing "stable" builds which turn out to be not that stable at all.

Cheers
manolito
Out of couriosity what happens when you remove _all_ CPP2.5 plugins then put them back one by one, which is the culprit?

Secondly
You can agree with FranceBB but you don't know what happened with his environment, nor do we know, but we'll solve it, maybe it was a mis-compiled version as I have written my example.

We can solve his problem with a much larger possibility that issues like why does not work properly and old tool which was tested and used when probably not even Avisynth 2.6 was final-released with plugins which I have to chase all around then finally download from Internet-wayback machine.
pinterf is offline   Reply With Quote
Old 23rd May 2020, 21:55   #204  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Quote:
BTW your script is using isb=false for both forward and backward vectors, which seems to be wrong.
Gah, a silly copy-paste mistake by me there in the example

Quote:
Originally Posted by pinterf View Post
Could you pls do a test: keep the original default 4GB memory and set the other cache mode, I wrote about above. What are your numbers now?
The default took about 5100 MB, SetCacheMode(1) went to ~4700 when the first frames of actual content (first 50 frames or so were just black) were being processed.

With max at 10000, the usage is almost 9300 MB with SetCacheMode(1).

I haven't done any real-life tests with encoding yet so I don't know what the optimal prefetch value is.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 24th May 2020, 08:13   #205  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Quote:
Originally Posted by pinterf View Post
A minor note: vector scaling is no longer necessary with newer mvtools, MDegraining a 16 bit clip will accept vectors made from arbitrary bit depth source. This is why you can MDegrain even a 32 bit float clip.
Out of curiosity, I am bit confused about how the HBD plugins treat the images.

AFAIK the most used bit depths nowadays are 8 and 10 (12 on some specialized streams only).

How does it work when you set HBD or a plugin receives a HBD?

Doest it switch to 16 bits (or more) regardless of true bit depth or does it work with the same bit depth of the source?

Because if it goes straight to 16 bit to work on a 10 bit stream, it seems to me really a waste of power and it would need a dedicated algorithm to process 10 bit only, as they are the second most common after 8, and get faster.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 24th May 2020, 08:20   #206  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Quote:
Originally Posted by pinterf View Post
1.) You can have now multiple Prefetchers (new in 3.6)
What is a good ratio to determine the number of frames to be prefetched? Now I am using Prefetch(6) because I am GPU memory bound. What effect has the frame parameter? Is it a sort of caching or what? Would set Prefetch (1,6) has the same effects on speed?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 24th May 2020, 08:32   #207  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
For internal calculations the bits used are usually more than that, 16-32 bits (15-16 bit is used even for most 8 bit filters) Only very simple filters, where no precise calculation happens, keep the bit depth internally (e.g. mt_binarize). Imagine, when you multiply two 8 bit data you'll have 16 bit result.
Processing 10 bits can be faster that 16 bits however, in mvtools I had differently optimized - quicker - SAD routines for 10 bits because summing up sixteen 10 bit numbers still fit in a 16bit register, while summing up 16bit numbers require 32 bit processing already.
But such optimization specialization is tricky, increases internal complexity and in general it is the very last step when developer have inifinite free time and cannot do other meaningful stuff
Plus: as it requires extra programming resources it has to be done when it has real benefits and the task to optimize is the bottleneck in a process.
pinterf is offline   Reply With Quote
Old 24th May 2020, 08:36   #208  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Quote:
Originally Posted by pinterf View Post
as it requires extra programming resources it has to be done when it has real benefits and the task to optimize is the bottleneck in a process.
I asked because I was planning to always use SMDegrain in HBD precision and then go back to 8 or 10 bit but sometimes the overhead is just too much to be of any use. Is there any optimization I could apply?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 24th May 2020, 08:38   #209  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Quote:
Originally Posted by tormento View Post
What is a good ratio to determine the number of frames to be prefetched? Now I am using Prefetch(6) because I am GPU memory bound. What effect has the frame parameter? Is it a sort of caching or what? Would set Prefetch (1,6) has the same effects on speed?
This parameter exists for your individual fine tuning and experiments. Probably when you find a number for a specific script or filter, you have to re-test it for another one. When you don't know if you need it, do not specify it.

But since we do not have knowledge base on the real value of this parameter, always share your findings.
pinterf is offline   Reply With Quote
Old 24th May 2020, 08:40   #210  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Quote:
Originally Posted by pinterf View Post
But since we do not have knowledge base on the real value of this parameter, always share your findings.
Can I have a wider explanation of what happens when I prefetch frames instead of threads? I have read 3.6.0 release notes but it did not clarify things a lot to me.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 24th May 2020, 08:40   #211  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Quote:
Originally Posted by tormento View Post
I asked because I was planning to always use SMDegrain in HBD precision and then go back to 8 or 10 bit but sometimes the overhead is just too much to be of any use. Is there any optimization I could apply?
Reduce your input to 320x200
pinterf is offline   Reply With Quote
Old 24th May 2020, 08:42   #212  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Quote:
Originally Posted by pinterf View Post
Reduce your input to 320x200
I deeply hate you with all my heart.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 24th May 2020, 09:52   #213  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Quote:
Originally Posted by tormento View Post
Can I have a wider explanation of what happens when I prefetch frames instead of threads? I have read 3.6.0 release notes but it did not clarify things a lot to me.
You are always specifying threads, then you can further specify the frame numbers to look ahead. Default value of frames is threads*2.
pinterf is offline   Reply With Quote
Old 24th May 2020, 12:48   #214  |  Link
gispos
Registered User
 
Join Date: Oct 2018
Location: Germany
Posts: 996
Quote:
Originally Posted by pinterf View Post
...Default value of frames is threads*2.
Thank you, I had already asked myself that.
Then my prefetch (4, 8) doesn't do anything for frame caching.
__________________
Live and let live

Last edited by gispos; 24th May 2020 at 12:51.
gispos is offline   Reply With Quote
Old 24th May 2020, 14:54   #215  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
The new cache mode has quite an impact at least in my case where I have that MVTools stuff + downscaling in linear light and debanding with neo_f3kdb. I tested with AVSMeter with an insanely high SetMemoryMax(16384) to make sure cache size will not restrict anything. The default cache mode ran at 9.495 fps and SetCacheMode(1) at 12.36 fps. Memory usage was almost the same, both around 11000 MB.

I'll continue testing to see if lowering the prefetch value from 24 makes any difference.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 24th May 2020, 16:41   #216  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Some tests of x265 encodes with different prefetch values and the abovementioned script structure:

threads=24, frames default 2.53 fps
threads=24, frames 12 2.57 fps
threads=24, frames 24 2.51 fps
threads=24, frames 8 2.56 fps
threads=24, frames 10 2.57 fps
threads=24, frames 16 2.55 fps
threads=22, frames 16 2.55 fps
threads=20, frames 12 2.58 fps
threads=16, frames 12 2.57 fps
threads=12, frames 12 2.55 fps

It's quite easy to see that most of the time is spent inside x265 instead of frameserving. At least in my case, lowering the amount of frames was slightly beneficial.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 24th May 2020, 16:52   #217  |  Link
stax76
Registered User
 
stax76's Avatar
 
Join Date: Jun 2002
Location: On thin ice
Posts: 6,837
In the staxrip tracker the most active power user posted when avisynth 3.5 is installed staxrip produces errors, it's fine if avisynth is not installed because then staxrip uses the included portable version 3.6 and it's also fine if 3.6 is installed. I had to explain this, maybe some devs can give some advice, my answer:

I changed it to don't allow old versions like currently done for Visual C++ 2019, it shows:

The currently used version of is not compatible (too old).

And it doesn't allow to edit the version except editing Versions.txt directly instead of using the GUI to do it.

Ideally it should not allow defining a custom path in this case but I'm not sure it's worth the trouble writing this code.

Here is the situation:

Most tools like StaxRip.exe or avs2pipemod.exe just load AviSynth with code like this:

LoadLibrary("AviSynth.dll")

This is however not a full path and what now is happening is the OS will search this DLL, for that it uses a strict order, first it will search it in the startup folder so next to staxrip.exe or next to avs2pipemod.exe or whatever tool is loading the avs file. Next it searches in the system folder, which is where avisynth is installed. I have tried to create soft links for avisynth.dll next to staxrip.exe but this did not work for unknown reason and then I gave up.

One issue was if avisynth is not installed then ffmpeg does not work because it does not use default DLL loading which searches in the path env var which staxrip sets. To fix that a soft link is working, qyot27 and stainless did help me to make this work.

In staxrip I could use a full path to load avisynth.dll and avs2pipemod has an option for that too, for the other tools an option could be requested, I can image some authors like rigaya would help, for some tools a mod could be created but for some tools it would be very difficult like ffmpeg.

Maybe it's not worth the trouble and just leave it as is, it's many avs tools like 10.

For vapoursynth it's working that a user can decide via setting if portable or installed version should be used, it can be defined in the settings dialog of staxrip and it worked for all tools I have tested, staxrip, ffmpeg, vspipe, mpv.net, only problem was mpc-be, not particular important since mpv.net is the primary player in staxrip and it supports portable mode for avisynth and vapoursynth. I have reported it and volt has a new avs vpy source filter in development which is awesome. Maybe somebody can try nvenc and qsvenc for which I don't have hardware or driver.

Last edited by stax76; 24th May 2020 at 17:07.
stax76 is offline   Reply With Quote
Old 24th May 2020, 17:00   #218  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Quote:
Originally Posted by qyot27 View Post
Still access violation.

Quote:
Originally Posted by pinterf View Post
This is weird, - I'm using cmakegui, checked the XP support checkbox, generated the solution file and it listed that v141_xp toolset is forced, OK.
Went into VS2019 GUI, opened project properties and there was no sign of XP toolset settings, only the /Zc:threadSafeInit- option. It still listed the non-XP v142 toolset.
EDIT: When I pressed the "Generate" once again then it created the proper xp targeted solution
I see! Would you mind uploading the newly created build?
Or is it the AviSynth+_3.6.0_i686-xp-nosimd uploaded by qyot27?

Quote:
Originally Posted by pinterf View Post
You can agree with FranceBB but you don't know what happened with his environment, nor do we know, but we'll solve it, maybe it was a mis-compiled version as I have written my example.

We can solve his problem with a much larger possibility
Well I'm not sure what happened with my environment either, but thank you for being confident that you can solve the issue. I really appreciate that.
FranceBB is offline   Reply With Quote
Old 24th May 2020, 19:27   #219  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Quote:
Originally Posted by pinterf View Post
Out of couriosity what happens when you remove _all_ CPP2.5 plugins then put them back one by one, which is the culprit?
Thanks for this suggestion, it helped me to narrow it down to the latest Manao MaskTools2 version 2.0a48. I use mt_masktools-26.dll in my AVS plugins folder, but for some reason AVSMeter recognized it as an AVS 2.5 plugin. Removing this DLL and replacing it with a current PinterF version solved the LAV Filters problem, but this is something I can't and do not want to do. (Replacing the DLL with the TP7 version 2.0 b1 made no difference.)

The problem is that several older AVSI scripts in my plugins folder rely on MaskTools2, and just replacing Maskools with the latest High Bit Depth versions will not work. A while ago I asked Real.Finder explicitly if I could use his latest Srestore version with an older version of MaskTools2, and his answer was NO. I guess the same is true for QTGMC or LFSMod or FineSharp. I do not need and do not want Hi Bit Depth or Hi Color, so I will stay away from such plugins. I installed AVS+ solely in the hope for better speed from multitasking, everything else has no relevance for me.

And if AVS+ 3.6 now gives me issues using older AVS 2.5 plugins then I will stop using it.

Cheers
manolito
manolito is offline   Reply With Quote
Old 25th May 2020, 05:31   #220  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Quote:
Originally Posted by manolito View Post
Thanks for this suggestion, it helped me to narrow it down to the latest Manao MaskTools2 version 2.0a48. I use mt_masktools-26.dll in my AVS plugins folder, but for some reason AVSMeter recognized it as an AVS 2.5 plugin. Cheers
manolito
Hi,
I checked it, and AvsMeter recognizes mt_masktools-2.6.dll a CPP2.5 DLL, probably because it is using the non-final AVS 2.6 header file, which has AVISYNTH_INTERFACE_VERSION = 5
instead of 6. So this is a pre-2.6 DLL.

EDIT: even if it used a pre-avs 2.6 DLL, the plugin doesn't have AvisynthPluginInit3, only AvisynthPluginInit2
EDIT: test-test-test Retro masktools 2.0.48 as an Avisynth 2.6 dll. https://drive.google.com/open?id=1xP...bT0JkffmEUQA67
Do _NOT_ mix it with masktools2!

Last edited by pinterf; 25th May 2020 at 11:38.
pinterf is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 02:13.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.