Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 16th April 2024, 23:06   #2821  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,602
Do we still need to declare SetMemoryMax?

I mean, almost 70% of the computers are now 64-bit with gazillions of GBs of RAM.

Is it of any use? Can't we simply let modern Windows to manage memory?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 16th April 2024, 23:20   #2822  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,827
You can probably omit it quite safely. Avisynth uses a convenient default.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 17th April 2024, 04:47   #2823  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,747
The default is quite conservative and is sometimes not enough if you work on 4K sources with heavy filtering and use multithreading. There's no real harm setting the max higher if you have the memory available, Avisynth will use what it needs and not all of it automatically.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 17th April 2024, 12:16   #2824  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,602
Perhaps I didn't explain well.

I know I can set it as I want but, seen VaporSynth memory use, why can't AVS+ go that way too and send Setmemorymax to ancient memories?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 17th April 2024, 12:17   #2825  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,602
Quote:
Originally Posted by Boulder View Post
The default is quite conservative and is sometimes not enough if you work on 4K sources with heavy filtering and use multithreading.
I can't find my post but, some years ago, I did some tests where I saw performance decrease with Setmemorymax increase.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 18th April 2024, 00:09   #2826  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,934
Quote:
Originally Posted by tormento View Post

I know I can set it as I want but, seen VaporSynth memory use, why can't AVS+ go that way too and send Setmemorymax to ancient memories?
It's already ancient memory.
There's no need to set it, Avisynth will manage that automatically.
By default, in modern x64 systems, it will use 4GB of RAM which is plenty.
Remember that this is the memory of the cache, in other words the memory used to store frames without having to fetch them again, NOT the RAM used by filters in general. This means that even if you were to set SetMemoryMax(512) but you were heavily filtering a UHD content, Avisynth would still use way more than 512MB of RAM. Remember that the cache is there only so that if you have temporal filters and encoders that need to access frames in a non linear fashion, Avisynth won't have to compute them every single time it moves forward and backward and forward and backward etc. The bigger the cache the more frames are gonna be stored, but storing too many frames can be detrimental as it's gonna use RAM that could otherwise be used by - let's say - the encoder, which is why the cache size is calculated on the fly on the basis of the available RAM and it never exceeds 4GB.

Assuming RGB48 (which is overkill anyway), we have:

3840216048 = 398131200 bits = 0.0498 gigabyte

That's just for one frame, so at the current default Avisynth can cache more than 80 frames. Obviously working with RGB48 UHD is a bit unrealistic, but if we work in YUV that number increases and it even goes much further if we work in 8bit and at lower resolutions.
My point is that it's already implemented correctly and I don't see a reason to change that. If anything, having the SetMemoryMax() is useful if you wanna reduce the cache.
FranceBB is offline   Reply With Quote
Old 18th April 2024, 04:20   #2827  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,131
It looks you not understand how frame-based AVS+ MT is working.

To make things as fast as possible (to minimize CPU stall on threads sync) it looks AVS+ simply put several frames cache around _each_ filter in the filtergraph and in _each_ logical thread. So any time system have some free logical CPU cores it can load it with some useful computing.

But it cause awful RAM consuming in any 'complex' scripts of a several filters in a chain.

Total RAM in a cache is about NumFrames_in_Prefetch_x_NumFilters_x_NumThreads_x_FrameSize.
DTL is offline   Reply With Quote
Old 18th April 2024, 04:59   #2828  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,747
Regarding the 4GB, which is unfortunately not enough in many cases: https://forum.doom9.org/showthread.p...75#post1913375
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 18th April 2024, 10:28   #2829  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,602
Quote:
Originally Posted by Boulder View Post
Regarding the 4GB
AVSMeter can provide realistic results?

(Where on hell has Groucho2004 gone?)

Secondo question: SetCacheMode(1) is a good choice?
__________________
@turment on Telegram

Last edited by tormento; 18th April 2024 at 10:30.
tormento is offline   Reply With Quote
Old 18th April 2024, 10:57   #2830  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,934
Quote:
Originally Posted by Boulder View Post
Regarding the 4GB, which is unfortunately not enough in many cases: https://forum.doom9.org/showthread.p...75#post1913375
Another interesting thing about prefetch, I see!


Quote:
Originally Posted by DTL View Post
Total RAM in a cache is about NumFrames_in_Prefetch_x_NumFilters_x_NumThreads_x_FrameSize.
I see! But to clarify I don't actually use Prefetch, like ever, I'm on the good old concept of letting plugin developers handle multithreading, so my scripts don't have it.
For instance, my scripts are like:

Code:
video=LWLibavVideoSource("video.mxf")
audio=LWLibavAudioSource("audio.mxf")
AudioDub(video, audio)

ConvertBits(16)

ConvertYUVtoXYZ(Color=0, OutputMode=1, HDRMode=0, fullrange=false)

ConvertXYZ_Reinhard_HDRtoSDR(exposure_X=2.5, contrast_X=0.9)

ConvertXYZtoYUV(pColor=0)

without prefetch. In the example above, the filters would create their own threadpool which is how I think it's intended to be. Call me traditionalist but I see Prefetch() as an evolution of the old MT Modes from 2009 and in my head they should only be used if the filter you're calling is old and single-threaded only and of course at your own risk as it might misbehave, especially if it filters temporally.


This however brings me to a different question, then.
Is there an advantage in raising the SetMemoryMax() value from 4GB to something higher IF I don't use Prefetch()?

Last edited by FranceBB; 18th April 2024 at 11:00.
FranceBB is offline   Reply With Quote
Old 18th April 2024, 12:08   #2831  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,131
If you not use Prefetch() it mean AVS+ will run all filtergraph with single threaded cache mode ? So the equation for total cache size is about
NumFrames_in_Prefetch_x_NumFilters_x_FrameSize.

Default num frames in AVS+ cache (for each filter) is at least several ? (2..3 to 10 ? or num of physical cores ?).

"Is there an advantage in raising the SetMemoryMax() value from 4GB to something higher IF I don't use Prefetch()?"

It may depends on workflow. And easy tested between 4 GB and all RAM avaialble. The idea of SetMemoryMax() may be complex hint to AVS memory manager like to save from swapping of too much allocated cache pages attempt to lower number of cached frames if total process RAM allocation approaches the current RAM limit.

Because with increasing number of frames in the cache user may got some performance boost (depending of filters used and many more effects) setting too low MemoryMax may cause decreasing of performance in some use cases. It is better to check with exact end user workflow.

Last edited by DTL; 18th April 2024 at 12:15.
DTL is offline   Reply With Quote
Old 18th April 2024, 12:58   #2832  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,747
Quote:
Originally Posted by tormento View Post
AVSMeter can provide realistic results?

(Where on hell has Groucho2004 gone?)

Secondo question: SetCacheMode(1) is a good choice?
Windows Task Manager shows the same information. With heavy prefetch settings like threads=32, frames=12 on my 5950X, a 4K source with MVTools based filtering and downscaling with avsresize's functionalities, outputting 16-bit data to x265 can take 20GB of memory. The amount of frames to prefetch is the key to memory usage. That's why it is important to restrict it if you have a lot of cores, the default is way too much (would be 64 frames in this case )

I've found out that mode 1 is better.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 18th April 2024, 13:01   #2833  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,747
Quote:
Originally Posted by FranceBB View Post
This however brings me to a different question, then.
Is there an advantage in raising the SetMemoryMax() value from 4GB to something higher IF I don't use Prefetch()?
I would say probably not. I've been running my chunked encoding tool and the memory usage is usually max 3GB per avs2yuv64 process with my standard denoising and resizing functions. Though it's with 10-bit output, I've not tested if 16-bit output changes things that much to go over the 4GB limit.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 18th April 2024, 16:56   #2834  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,602
My findings.

Basic script

PHP Code:
SetFilterMTMode("DEFAULT_MT_MODE"2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("M:\In\2013 GIT - Arise ~864p Dynit\2013_06 Ghost pain.dgi")
Comptest24(1)
Convertbits(16)
libplacebo_Resample(1536,864,filter="ewa_lanczossharp")
SMDegrain (tr=3thSAD=300refinemotion=falsecontrasharp=falsePreFilter=7plane=4chroma=truemode="MDegrain")
libplacebo_Deband(iterations=3temporal=true)
fmtc_bitdepth (bits=10,dmode=7)
Prefetch(3
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4071, 3.7, x86_64) (3.7.3.0)

Number of frames: 1680
Length (hh:mm:ss.ms): 00:01:10.070
Frame width: 1536
Frame height: 864
Framerate: 23.976 (24000/1001)
Colorspace: YUV420P10

Frames processed: 1680 (0 - 1679)
FPS (min | max | average): 1.202 | 116279 | 11.54
Process memory usage (max): 1990 MiB
Thread count: 45
CPU usage (average): 35.7%

GPU usage (average): 15%
VPU usage (average): 8%
GPU memory usage: 1496 MiB
GPU Power Consumption (average): 41.6 W


x265 slow: encoded 1680 frames in 297.25s (5.65 fps), 1134.83 kb/s, Avg QP:23.77

Adding on top:

PHP Code:
SetMemoryMax() 
FPS (min | max | average): 1.589 | 55866 | 11.33
Process memory usage (max): 1947 MiB
Thread count: 45
CPU usage (average): 35.6%


x265 slow: encoded 1680 frames in 293.64s (5.72 fps), 1138.50 kb/s, Avg QP:23.70

Adding on top:

PHP Code:
SetCacheMode(1
FPS (min | max | average): 0.340 | 178572 | 11.52
Process memory usage (max): 1849 MiB
Thread count: 45
CPU usage (average): 35.7%


x265 slow: encoded 1680 frames in 297.86s (5.64 fps), 1137.39 kb/s, Avg QP:23.72

Adding on top:

PHP Code:
SetMemoryMax()
SetCacheMode(1
FPS (min | max | average): 0.338 | 133333 | 11.42
Process memory usage (max): 1846 MiB
Thread count: 45
CPU usage (average): 35.6%


x265 slow: encoded 1680 frames in 284.14s (5.91 fps), 1136.06 kb/s, Avg QP:23.72

I see some discrepancies between avs+ only and real world scenario with x265 encoding but, perhaps, is just my old machine.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 18th April 2024, 18:07   #2835  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,131
"SetMemoryMax() "

I think it requires memory size (in MBytes ?). Empty params call do not change current AVS memory max.

SetMemoryMax(amount)
Sets the maximum memory that AviSynth uses (in MB) to the value of amount. Setting to zero just returns the current Memory Max value. In the 2.5 series the default Memory Max value is 25% of the free physical memory, with a minimum of 16MB.
The default Memory Max is also limited to 512MB.
AVS+In Avisynth+ this limit for default Memory Max is 1024MB for 32 bits and 4096MB on the x64 version
DefaultMemoryMax = minimum(physical_memory / 4, secondary_memory_max_limit)

This really working:
Code:
ColorBars()

mem=SetMemoryMax(0)
SubTitle(Format("old default max={mem}!"))


SetMemoryMax(5000)
mem=SetMemoryMax(0)
SubTitle(Format("new max={mem}!"), align=4)
First call returns old memory max value (4074 at me) and second call sets to 5000 MB.

So you can first check your current system default memory max and check some significant changes like set to 10 times lower and about 70..80% of current RAM installed.

Last edited by DTL; 18th April 2024 at 18:16.
DTL is offline   Reply With Quote
Old 18th April 2024, 18:54   #2836  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 1,028
Where did this come from ??

AviSynth+ 3.7.3 (r4071, 3.7, x86_64) (3.7.3.0)
__________________
Long term RipBot264 user.

RipBot264 modded builds..
*new* x264 & x265 addon packs..
TDS is offline   Reply With Quote
Old 18th April 2024, 20:30   #2837  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,131
From documentation at http://avisynth.nl/index.php/Internal_functions

http://avisynth.nl/index.php/Interna...s#SetMemoryMax
DTL is offline   Reply With Quote
Old 18th April 2024, 22:27   #2838  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 1,028
Sorry DTL, you misunderstood

Where did that build of Avisynth come from, (r4071) In tormento's post.

But

Found it....https://gitlab.com/uvz/AviSynthPlus-Builds
__________________
Long term RipBot264 user.

RipBot264 modded builds..
*new* x264 & x265 addon packs..

Last edited by TDS; 19th April 2024 at 01:40. Reason: found it
TDS is offline   Reply With Quote
Old 19th April 2024, 12:21   #2839  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,602
Quote:
Originally Posted by DTL View Post
"SetMemoryMax() "

I think it requires memory size (in MBytes ?). Empty params call do not change current AVS memory max
I look at numbers

Something happens and is not what I expected.

Some years ago I did benchmarks here with various amount of memory and that () lead to the best results, even against higher ones.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 23rd April 2024, 05:59   #2840  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,835
Is it intended for the Convert functions to change the chroma location when they're not actually changing the color format?

For example, if you have a YUV420 source with the chroma location in frame properties as "top_left", and you add ConvertToYUV420() to the script, the color format doesn't change but the chroma location changes to "left".

I know "left" is the default output chroma location, and it works as the Avisynth wiki says it does, but before I noticed the chroma location had changed and checked, I expected it to remain the same unless the color format had changed.

I can't decide if changing the location when there's no conversion taking place is a good idea. Or maybe it doesn't matter much....

Cheers.

Last edited by hello_hello; 24th April 2024 at 12:52.
hello_hello is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 05:53.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.