Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 30th August 2016, 18:39   #2361  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 1,703
Quote:
Originally Posted by Groucho2004 View Post
I just found it odd that you put such a high value considering the low memory requirements of that particular script.

You may have to increase the default (512 MB) if you have a script with a large temporal range but you would test with small increments, 256 ~ 512.


It comes from my SMDegrain 6 frames span script + KNLMeans.

No idea of how AviSynth+ x64 deals with memory. I have plenty of it. I read that too much SetMemory could be negative too but it referred to old AviSynth. If any of the active devolver could clarify this aspect, would be very useful.
__________________
@turment on Telegram
tormento is offline  
Old 30th August 2016, 18:56   #2362  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 765
Tried to wrap VD filter through avs with avsplus-r1858-pfmod

Quote:
Avisynth open failure:
rgb_levels has an invalid parameter string (bug in filter)
the script:
Quote:
AviSource("E:\vd_dev\data\cfhd-high.avi")
LoadPlugin("E:\vd_dev\vd\VDubFilter.dll")
LoadVirtualdubplugin("E:\vd_dev\vd\plugins32\rgb_levels.vdf", "rgb_levels")
filter (nothing really special): https://sourceforge.net/projects/vdf...s/version%202/

arg_list from filter definition: "ddddddddd" (9 floats)

What`s wrong?
__________________
VirtualDub2
shekh is offline  
Old 30th August 2016, 19:48   #2363  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
The memory and cache management in Avs+ is completely different from Avs 2.6. It has been redesigned and rewritten from scratch, though I was uncomfortably constrained by the existing API which resulted in half-assed code for memory reclamation in some (hopefully not too common) cases. Anyway, the caches in Avs+ are highly adaptive and self-learning, and do not use any caching hints like 2.6 in general to perform their jobs well. Also, as long as there is enough memory to satisfy all memory requests, even uncapped caches should not cause significant slowdowns, because caches will only grow if needed, not as long as there is free memory.

The default setting to SetMemoryMax() can be up to 1GB in Avs+, but may be lower depending on your amount of RAM. I know in earlier Avs 2.6 alpha/2.5 it was necessary to keep SetMemoryMax() low because if too high it resulted in funny caching behavior, leaving users with phenomena like slowdowns and/or unreasonably high memory usage. Avs+ should not suffer from anything like that, and SetMemoryMax() really just sets an (approximate) upper cap. If your script does not need that much memory though, a high upper limit (even if too high) does zero harm. In Avs+, the only use of constraining memory is to prevent paging. The justification is that often it is faster to recompute frames than to trash your rotational paging HDD (ofc if you have an SSD you'll have to reevaluate this claim). Another (similar) justification is, even if you had enough RAM, encodes often take many hours and sometimes even days, and if you want to keep using your computer in the background without hitting the paging limit, you might want to constrain the memory consumption of the encoding process.

But this is all TL;DR. The important takeaway is that in Avs+, a high SetMemoryMax() only hurts if otherwise your script causes excessive paging to a slow hard disk. Pro tipp: If you enable logging in Avs+, it will automatically notify you if you can speed up your script by setting your SetMemoryMax() higher Edit: The optimal setting of SetMemoryMax() in Avs+ is anywhere in the range where it is higher than what your script wants to use, but lower than the value where it would cause paging. This range is often very broad, so only those will have trouble setting it who have either low RAM, or extremely hungry scripts. This is just a "rule of thumb" though, and its applicability will depend on your script and hardware.
__________________
AviSynth+

Last edited by ultim; 30th August 2016 at 19:56.
ultim is offline  
Old 30th August 2016, 20:02   #2364  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,099
Speaking of which, does Avs+ reuse framebuffers like the OG Avisynth does?
TheFluff is offline  
Old 30th August 2016, 20:40   #2365  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by TheFluff View Post
Speaking of which, does Avs+ reuse framebuffers like the OG Avisynth does?
Depends. Probably. Dunno what you mean exactly by "reuse".
__________________
AviSynth+
ultim is offline  
Old 30th August 2016, 20:49   #2366  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by rean View Post
I see AviSynth+ is based on the memory allocation code of legacy AviSynth...
Nope, not at all. See above.
__________________
AviSynth+
ultim is offline  
Old 30th August 2016, 21:34   #2367  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,099
Quote:
Originally Posted by ultim View Post
Depends. Probably. Dunno what you mean exactly by "reuse".
(this is from memory, might be completely wrong)
OG Avisynth refcounted vfb's and when the refcount reached zero, instead of freeing the buffer it was kept around and a subsequent call to NewVideoFrame for a suitably sized frame could use that buffer instead of allocating a new one.

I'm asking because as far as I know VS didn't bother with this and I seem to remember Myrsloik stating that simple free/new malloc for "small buffers" worked "well enough". I'm curious how much the buffer reuse technique helps, especially for larger video frame sizes.

Last edited by TheFluff; 30th August 2016 at 21:36.
TheFluff is offline  
Old 30th August 2016, 22:01   #2368  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by TheFluff View Post
(this is from memory, might be completely wrong)
OG Avisynth refcounted vfb's and when the refcount reached zero, instead of freeing the buffer it was kept around and a subsequent call to NewVideoFrame for a suitably sized frame could use that buffer instead of allocating a new one.

I'm asking because as far as I know VS didn't bother with this and I seem to remember Myrsloik stating that simple free/new malloc for "small buffers" worked "well enough". I'm curious how much the buffer reuse technique helps, especially for larger video frame sizes.
Yes that technique is still there, and admittedly I intend to keep it around. Not because of some speed argument, but because it improves reliability. Simply put, it prevents memory fluctuation. It ensures that once Avs+ allocated a buffer, the same size can be allocated again after the previous instance is dereferenced. If Avs+ was to free each buffer whose refcount falls to zero, there'd be a chance that due to some other thread or application also allocating memory, Avisynth wouldn't be able to get a new buffer anymore, even though its total memory consumption didn't increase. This means without this technique there is a higher chance that your encode goes OutOfMemory in the middle, only because some other process or application thread had a sudden (maybe even temporary) memory spike.
__________________
AviSynth+
ultim is offline  
Old 30th August 2016, 22:11   #2369  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,054
Quote:
Originally Posted by ultim View Post
Yes that technique is still there, and admittedly I intend to keep it around. Not because of some speed argument, but because it improves reliability. Simply put, it prevents memory fluctuation. It ensures that once Avs+ allocated a buffer, the same size can be allocated again after the previous instance is dereferenced. If Avs+ was to free each buffer whose refcount falls to zero, there'd be a chance that due to some other thread or application also allocating memory, Avisynth wouldn't be able to get a new buffer anymore, even though its total memory consumption didn't increase. This means without this technique there is a higher chance that your encode goes OutOfMemory in the middle, only because some other process or application thread had a sudden (maybe even temporary) memory spike.
Thank you for that insight.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline  
Old 31st August 2016, 03:18   #2370  |  Link
bilditup1
Registered User
 
bilditup1's Avatar
 
Join Date: Feb 2004
Location: NYC
Posts: 124
DGDecode_mpeg2source no longer works in MT

Moving from r1858_pfmod to r2172, I've been getting the following error when attempting to preview a fairly simply script in AvsPmod:
Code:
"Only a single prefetcher is allowed per script." (path_to_script.avs, line 13)
This is the script, which isn't particularly complex:
Code:
Import("C:\blu\MeGUI\tools\avisynth_plugin\reel.deel\mt_modes_latest.avsi")
LoadPlugin("C:\blu\MeGUI\tools\dgindex\DGDecode.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\jpsdr\x86\Release_Intel_W7_Core2_SSE4.2\nnedi3.dll")	
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\RgTools.dll")			
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\MaskTools2.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\pinterf\MvTools2.7.0.22d.dll")		
Import("C:\blu\MeGUI\tools\avisynth_plugin\real_finder\QTGMC.avsi")
DGDecode_mpeg2source("d2v-path-here") # 1080i30 mpeg2 src
AssumeBFF()
QTGMC(Preset="Fast", ShowSettings=False)
Spline36Resize(1280,720)
Trim(0,3000)
Prefetch(4)
I checked the scripts I was importing - reel.deel's mt modes list and QTGMC - but Prefetch() is never called in them.
This error message must be a recent change, as in r2161, AvsPmod just crashes entirely, and when attempting to encode, the thread simply stalls and makes no progress (the same thing).
Removing the Prefetch call allows the script to be processed, in both r2161 and r2172, but I don't think mt is invoked, as it runs at st speeds (6.5fps vs 8.91).
I thought this was a problem with jpsdr's nnedi3, which I had to update to r25/r26 from r22 in order to avoid a memory violation with these new releases. But I reduced to script to merely this:
Code:
LoadPlugin("C:\blu\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("d2v-path-here") # 1080i30 mpeg2 src
Prefetch(4)
And still get the same error. Is this somehow a quirk of dgdecode? Is mt broken? Is it enabled differently now? Or is something else at work here?

ED: It is in fact DGDecode.dll. Using DGDecodeIM.dll and a dgi instead did not present this error - not only that but the encode goes some 50% faster (from 9 to 13.5 fps) than the original script did under r1858_pfmod and DGDecode (though the first time I tried it, it quit after one frame with a typical memory access error.)
Does anybody know what's going on with DGDecode? I guess it could be considered legacy software by now, and presume that maintaining compatibility with it probably is not a priority, but this strikes me as odd behavior all the same.

ED2: Part of the higher speed and memory access errors I had been experiencing appears to be because I had erroneously switched to using the XP compatible build of nnedi3. The W7_SSE4.2 build still goes faster than before but less dramatically (12fps instead of 13.5fps, and trying the XP build again just now I only managed to reach 12.6fps, weirdly), and you pay for it with the danger of a memory problem. I've also experienced what I believe are memory issues when using nnedi3 with jpsdr's ResampleMT (latest) - iow encode just stops in the middle but without quitting or throwing an error, just sticks at a certain frame - though not consistently: when it does work then it brings the speed back up to 13.5 even whilst using the W7_SSE4.2 builds of both it and nnedi3. (Will also post about it in jpsdr's threads, I suppose, if someone hasn't already pointed this out.)

ED3: Groucho2004's build of DGDecode has the same problem as neuron2's.

Last edited by bilditup1; 31st August 2016 at 05:31. Reason: ED2
bilditup1 is offline  
Old 31st August 2016, 05:42   #2371  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by bilditup1 View Post
Moving from r1858_pfmod to r2172, I've been getting the following error when attempting to preview a fairly simply script in AvsPmod:
Code:
"Only a single prefetcher is allowed per script." (path_to_script.avs, line 13)
This should be a regression in Avs+ in the recent build. I'll take care of it by the weekend.
__________________
AviSynth+
ultim is offline  
Old 31st August 2016, 06:32   #2372  |  Link
bilditup1
Registered User
 
bilditup1's Avatar
 
Join Date: Feb 2004
Location: NYC
Posts: 124
Quote:
Originally Posted by ultim View Post
This should be a regression in Avs+ in the recent build. I'll take care of it by the weekend.
Wow, thanks for quick reply. Looking fwd to fix. Unfortch I've just discovered that TDecimate does not work with MT enabled, whether using tritical's original TIVTC.dll or groucho2004's build. Works fine in r1858_pfmod, quits with "internal error during prebuffering!" in r2085, r2161, r2172.
Attached Images
  

Last edited by bilditup1; 31st August 2016 at 06:35. Reason: spelling
bilditup1 is offline  
Old 31st August 2016, 08:34   #2373  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 1,703
Quote:
Originally Posted by ultim View Post
The default setting to SetMemoryMax() can be up to 1GB in Avs+
So, having 16+ GB of RAM and a x64 environment, which is the maximum value we can set?

Is it of any use or we can write scripts without allocating memory?
__________________
@turment on Telegram
tormento is offline  
Old 31st August 2016, 08:48   #2374  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,054
Quote:
Originally Posted by bilditup1 View Post
I've been getting the following error when attempting to preview a fairly simply script in AvsPmod:
...
This is the script, which isn't particularly complex:
Code:
Import("C:\blu\MeGUI\tools\avisynth_plugin\reel.deel\mt_modes_latest.avsi")
LoadPlugin("C:\blu\MeGUI\tools\dgindex\DGDecode.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\jpsdr\x86\Release_Intel_W7_Core2_SSE4.2\nnedi3.dll")	
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\RgTools.dll")			
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\MaskTools2.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\pinterf\MvTools2.7.0.22d.dll")		
Import("C:\blu\MeGUI\tools\avisynth_plugin\real_finder\QTGMC.avsi")
DGDecode_mpeg2source("d2v-path-here") # 1080i30 mpeg2 src
AssumeBFF()
QTGMC(Preset="Fast", ShowSettings=False)
Spline36Resize(1280,720)
Trim(0,3000)
Prefetch(4)
The script is far from simple, have a look at the content of qtgmc.avsi. As for your stability problems (aside from a AVS+ issue), see also here.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline  
Old 31st August 2016, 08:56   #2375  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 765
Quote:
Originally Posted by shekh View Post
Tried to wrap VD filter through avs with avsplus-r1858-pfmod

Avisynth open failure:
rgb_levels has an invalid parameter string (bug in filter)
...
same with 2172
shekh is offline  
Old 31st August 2016, 11:27   #2376  |  Link
burfadel
Registered User
 
Join Date: Aug 2006
Posts: 2,234
Ultim, I hope I wasn't 'out-of-line' PM'ing you earlier with the script sample, I just wanted to avoid stoking the fire of the other threads regarding upsampling, changing resolutions etc., seeing as the script did involve some manipulation in that regard. I hope that it does prove useful though in what I mentioend regarding single thread and multithreaded modes etc, and also how it can run quite fast.

Is my assumption wrong in that any script either becomes all multithreaded on the avisynth side, even with MT mode 3 on the plugins, or single threaded on the Avisynth side when prefetch is not stated, but still multithreaded invidividual plugins if the plugins are multithreaded? If the former isn't right, it does seem to exhibit that behaviour.
burfadel is offline  
Old 31st August 2016, 11:49   #2377  |  Link
bilditup1
Registered User
 
bilditup1's Avatar
 
Join Date: Feb 2004
Location: NYC
Posts: 124
Quote:
Originally Posted by Groucho2004 View Post
The script is far from simple, have a look at the content of qtgmc.avsi.
Yes, I'm aware that QTGMC itself is complex. I meant that my snippet was legible, straightforward and didn't have much of my own more-likely-to-be-faulty logic in it. In any case, QTGMC's presence was ultimately irrelevant.
bilditup1 is offline  
Old 31st August 2016, 18:52   #2378  |  Link
Wilbert
Moderator
 
Join Date: Nov 2001
Location: Netherlands
Posts: 6,341
Quote:
Originally Posted by tormento View Post
What Groucho2004 said. You have to be patience. I imported your script in Virtualdub and compressed it to Xvid but it took me 10 hours on my slow laptop (2.2 GHz). I used plain AviSynth though, but that probably doesn't matter. Btw, it consists of 324 images (iirc).
Wilbert is offline  
Old 31st August 2016, 22:37   #2379  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,054
Quote:
Originally Posted by Wilbert View Post
Btw, it consists of 324 images (iirc).
Should be 434.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline  
Old 1st September 2016, 00:04   #2380  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,268
Quote:
Originally Posted by AzraelNewtype View Post
There is nothing ordinary about a nearly CD sized animated gif.
I've noticed this happening on image hosting sites that offer a gif version and a webm version of whatever the animated image sequence is. The gifs in those cases are sometimes outlandishly sized, because I assume what's going on is the image host offers the ability to upload video, so the user uploads a video file, and then the image host compresses it to webm and to gif. But the gif's resolution hasn't been reduced, so you end up with ridiculous 1920x1080 animated gifs alongside a smaller, higher quality normal video file.

I keep asking myself what the use case of that could possibly be vs just using a video host, but I've never come up with a satisfactory answer. Because even meme sites like these gif posts probably are intended for should really have no issues embedding video files in 2016, and if your computer is so old that it can't decode one of those VP8 or VP9 webm files, is your browser going to somehow not choke on a 1080p gif?
qyot27 is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 07:43.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.