RipBot264 v1.18.3 - Simple and easy to use GUI -> IPOD . PSP . CONSOLES . BLURAY

Originally Posted by Atak_Snajpera

Or changing SetCacheMode
http://avisynth.nl/index.php/Interna...s#SetCacheMode

Guys add that at the top of your script and see if this fixes that performance drop on 7850 with 16 prefetch threads with active MDegrain/SMdegrain

SetCacheMode(1)

Be nice if this was the "magic" formula...will definitely test this tomorrow.



EDIT:- What a shame it wasn't

[rlev11]

Originally Posted by TDS

So using the exact same test file as the 7950X tests.

Tested a 5 minute 4K clip, using 1 x 6 minute chunk size, CQ 16, no filters !!
using this x265 Profile:-

--level 6.2 --profile main10 --hdr10 --hdr10-opt --output-depth 10 --ctu 64 --high-tier --vbv-bufsize 800000 --vbv-maxrate 800000 --asm avx512 (note:- the avx512 command is ignored by the 13900KF)

/avisynth-prefetch-threads 12 /x264-threads 16 /x265-threads 16 Time Taken 10:07 @ 11.88fps

/avisynth-prefetch-threads 14 /x264-threads 16 /x265-threads 16 Time Taken 10:04 @ 11.95fps

/avisynth-prefetch-threads 16 /x264-threads 16 /x265-threads 16 Time Taken 10:39 @ 11.29fps


but then I thought I'd try some Affinity tests (no prefetch) :-

/affinity 3FFC3FFC (which equates to 12 threads, but displays as 24 in the Encoder Server window) Time Taken 08:29 @ 14.16fps

/affinity 3FFFFFFC (which equates to 14 threads, but displays as 28 in the Encoder Server window) Time Taken 08:15 @ 14.57fps

/affinity FFFFFFFF (which equates to 16 threads, but displays as 32 in the Encoder Server window) Time Taken 08:04 @ 14.92fps (fastest)

then no special "instructions", (which equates to 16 threads, but displays as 32 in the Encoder Server window) Time Taken 08:06 @ 14.85fps

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

So what I would now like to know is, if the 7950X was the Client machine, to the 13900KF being a Server, do the Encoder command line settings get transferred

from the Client to the Server, meaning that the above settings would revert back to :-

/avisynth-prefetch-threads 12 /x264-threads 16 /x265-threads 16.

OR

Does each PC use it's own particular Encoder settings ??

So it appears that Affinity has a very different influence on the 13900KF, over 16 core Ryzen's.

I guess the next "test" is a proper encoding job, with 2 or so DE Servers working.

so I set the client machine encoding servers in the ripbot264.ini by just using this:
Server1CommandLine=/port 1000 /priority normal /restart-if-no-progress /avisynth-prefetch-threads 12 /x264-threads 16 /x265-threads 16
Server2CommandLine=/port 2000 /priority normal /restart-if-no-progress /avisynth-prefetch-threads 12 /x264-threads 16 /x265-threads 16

All my encoding servers are set by a windows batch file that sits on the desktop and also a shortcut is added to the startup folder so when a server is booted, it loads automatically and is ready to go. I have this in the /bat file for the 16 core Ryzens:
encodingserver /port 1000 /priority normal /restart-if-no-progress /avisynth-prefetch-threads 12 /x264-threads 16 /x265-threads 16

and just this in the 12 and 10 core boxes:
encodingserver /restart-if-no-progress /priority normal

So as far as I know the client does not "push" its encoding server settings to the servers

[rlev11]

Originally Posted by Atak_Snajpera

Or changing SetCacheMode
http://avisynth.nl/index.php/Interna...s#SetCacheMode

Guys add that at the top of your script and see if this fixes that performance drop on 7850 with 16 prefetch threads with active MDegrain/SMdegrain

SetCacheMode(1)

I want to setup my somewhat controlled test that I did over this past weekend to really look at this which I will get to later in the week.

My initial "way too early" observation is that it does make a difference, but I am not sure it is the magic pill yet. On the 7950x and 5950x the utilization graph is a lot more stable across the board, not the constant wild swings in the decoder utilization. There are still some points during the middle of a chunk where the encoder drops out and the decoder takes a dive for 10-15 seconds. Then it goes back to a stabilized somewhat flat bar. Running at 12 core the bar is stable the whole way across the chunk. FPS are definitely better coming close to the 5900x numbers, but not as high as they should be.

One thing I did notice was I opened a second encoding window on the 7950x at full cores, and performance dropped a little bit combined vs single. When I did this on the 5950x, both encoding windows puked their guts out.

Will post back when I look more thoroughly at it later this week

Originally Posted by rlev11

so I set the client machine encoding servers in the ripbot264.ini by just using this:
Server1CommandLine=/port 1000 /priority normal /restart-if-no-progress /avisynth-prefetch-threads 12 /x264-threads 16 /x265-threads 16
Server2CommandLine=/port 2000 /priority normal /restart-if-no-progress /avisynth-prefetch-threads 12 /x264-threads 16 /x265-threads 16

All my encoding servers are set by a windows batch file that sits on the desktop and also a shortcut is added to the startup folder so when a server is booted, it loads automatically and is ready to go. I have this in the /bat file for the 16 core Ryzens:
encodingserver /port 1000 /priority normal /restart-if-no-progress /avisynth-prefetch-threads 12 /x264-threads 16 /x265-threads 16

and just this in the 12 and 10 core boxes:
encodingserver /restart-if-no-progress /priority normal

So as far as I know the client does not "push" its encoding server settings to the servers

I know I should have known about this, having used RB long enough.

I guess I just wanted another user's confirmation.

Thanks

Now onto more testing..

So I did a lot more testing earlier today, and posted a rather large page of info.

But then I started to do some "proper" encodes, and the whole process went out the window, it just wasn't relevant, so I deleted the post

You can test all you like, and maybe the "test" file isn't too complex, so it doesn't really reflect what you will encounter with any other encode.

I added SetCacheMode(1) to the script of the encode I was going to do, and tbh, I don't think it makes much, if any difference, but having said that, it might as well added to any "custom" script, just in case.

So all the info I have posted over the last couple of days if kinda moot. , and I might even delete it..we'll see.

Anyway, I did a 4K encode with some different settings, and the 13900KF was actually faster than the 7950X (for the first time), but then when I went onto the next job, the 13900KF "fell off a cliff", and the 7950X , "went to the moon".

So about all I can say is I am currently using these command line's :-

/avisynth-prefetch-threads 12 /x264-threads 16 /x265-threads 32

for the 7950X, & 3950X.

&

/avisynth-prefetch-threads 16 /x264-threads 16 /x265-threads 32

for the 13900KF.

I even tried this x265 command :-

--level 6.2 --profile main10 --hdr10 --hdr10-opt --output-depth 10 --ctu 64 --high-tier --vbv-bufsize 800000 --vbv-maxrate 800000 --asm avx512 --pools 32

Note:- as I haven't been doing any x264 stuff, that number hasn't changed, but it might.

[rlev11]

I ran through the same tests and setup as this past weekend except this time i added the "SetCacheMode(1)" to the top of the avisynth script.

What I can basically say is that it made little to no difference on any of the tests as before. If anything the 5950x's trended a little better and the 7950x trended a little worse, but all by very small percentages. This was for both the 4k and 1080p runs

It definitely did not take care of the issue with 4k encodes running full out on the 5950x and 7950x, but it was worth a shot trying it, thanks...

I did one last test on the 4k Animal House, this time not using any avisynth script or settings, just load it and go. This results in no decoding going on and just showing encoder cpu time. The 7950x running at all 16 cores was by far the fastest, followed by the 5950x at all 16 cores, and then I tried with the 7950x with 12 core using affinity mask and that was slower than the others which I expected.

This makes me convinced the issue has to be with the relationship between ffmpeg and it using the avisynth script as the input and with avisynth sending more information to ffmpeg than it can handle when the size of the stream gets to a certain point with 4k. The hose between them just isn't big enough and it may just mean that the hose can't get any larger.

[Boulder]

Originally Posted by TDS

Be nice if this was the "magic" formula...will definitely test this tomorrow.



EDIT:- What a shame it wasn't

Then check how much memory one Avisynth related process consumes. The thing is that by default, the amount of cache is restricted to 4GB for 64-bit Avisynth. When you set threads=x in Prefetch, Avisynth+ will then set frames=x*2 as prefetched frames by default. This can easily kill performance (by eating way too much memory + overhead is bigger) so it's better to set the amount of frames manually. For my 5950X, threads=32, frames=12 works best but with RB it's different if you launch multiple encodes on the same machine.

A 4K encode can easily take over 4GB of cache memory. Some scripts I've used, have taken almost 20GB You'll need SetMemoryMax to raise the amount if the default is not enough.

[Atak_Snajpera]

Originally Posted by rlev11

I ran through the same tests and setup as this past weekend except this time i added the "SetCacheMode(1)" to the top of the avisynth script.

What I can basically say is that it made little to no difference on any of the tests as before. If anything the 5950x's trended a little better and the 7950x trended a little worse, but all by very small percentages. This was for both the 4k and 1080p runs

It definitely did not take care of the issue with 4k encodes running full out on the 5950x and 7950x, but it was worth a shot trying it, thanks...

I did one last test on the 4k Animal House, this time not using any avisynth script or settings, just load it and go. This results in no decoding going on and just showing encoder cpu time. The 7950x running at all 16 cores was by far the fastest, followed by the 5950x at all 16 cores, and then I tried with the 7950x with 12 core using affinity mask and that was slower than the others which I expected.

This makes me convinced the issue has to be with the relationship between ffmpeg and it using the avisynth script as the input and with avisynth sending more information to ffmpeg than it can handle when the size of the stream gets to a certain point with 4k. The hose between them just isn't big enough and it may just mean that the hose can't get any larger.

Try now with

SetMemoryMax(16384)

[rlev11]

Tried earlier with SetMemoryMax(8192) along with the cache setting and saw no difference. My only question with that is where would I see the memory being used by avisynth. It does not show up in task manager anywhere that I can see. ffmpeg with and without the 8192 setting was at around 5.5 gig memory used in task Manager. I will try setting it to 16 tomorrow sometime. In the middle of something (work) at the moment. Only problem i could see if 16gig works is some of my servers only have 16 in them (the majority are 32) and that setting would to push out to all the servers since it is in the avisynth script (but we would be closer to solving the mystery).

Also Atak, can we and what would be the correct way to set the frames for avisynth referenced by Boulder above for some testing. Would it be "/avisynth-prefetch-threads 16 frames 16" or "/avisynth-prefetch-threads 16 /avisynth-prefetch-frames 16"

Originally Posted by rlev11

Tried earlier with SetMemoryMax(8192) along with the cache setting and saw no difference. My only question with that is where would I see the memory being used by avisynth. It does not show up in task manager anywhere that I can see. ffmpeg with and without the 8192 setting was at around 5.5 gig memory used in task Manager. I will try setting it to 16 tomorrow sometime. In the middle of something (work) at the moment. Only problem i could see if 16gig works is some of my servers only have 16 in them (the majority are 32) and that setting would to push out to all the servers since it is in the avisynth script (but we would be closer to solving the mystery).

Also Atak, can we and what would be the correct way to set the frames for avisynth referenced by Boulder above for some testing. Would it be "/avisynth-prefetch-threads 16 frames 16" or "/avisynth-prefetch-threads 16 /avisynth-prefetch-frames 16"

I have tried a couple of attempts of the "frames" setting, with no success.

Am currently trying SetMemoryMax(16384), & SetMemoryMax(24576), and this, SetMemoryMax(32768), failed, I only have 32Gb of system memory.

It's starting to be a PITA

EDIT:- So how I'm "testing" today is actually doing a "proper" encode, and stopping after each completed chunk, for adjustments, and I have tried different SetMemoryMax(), and it doesn't make any significant difference, sadly.
I think that it might be best practice to just run one DE Server per PC (which sounds like a waste), but the single chunk encoding speeds a really up there , I am using a 2 minute chunk, but it might be worth using 3 or 4 minute chunks, so the CPU has a chance to "get its teeth into it", and it would also "trick" us into thinking that it's not a big job, as there are way less chunks in the list.

[rlev11]

I've always used a chunk size of 2. The problem with going larger especially with distributed encoding is inevitably your slowest server or both chunks of a server running 2 encoders will end up being the last couple chunks to finish up. Add that to the last chunk sometimes being close to double it's size. An encoding job could end up being 20-30 minutes longer depending on how those last couple of chunks line up.

And that brings up another variable in all of this is total time for an encoding job. Just looking a FPS is only half the story. One could look at 2 similar length jobs and 1 could be significantly faster running 2 encoding windows per server, and the other could be significantly faster running just 1 encoding window per server depending on how things line up and there is no way to tell ahead of time. There is no way to really get around this, it is just the nature of the beast. Eventually I guess it would all even out.

Don't get me wrong, I am really pleased with the performance of the 59 and 79 50's once I get them setup properly. Right now this whole thing of why we have to knock down the 16 core Ryzens is more of an itch I keep scratching. Even though if we do eventually figure out what is going on, I doubt the performance increase would be all that much more than what we are getting now. It's more a curiosity now.

Originally Posted by rlev11

I've always used a chunk size of 2. The problem with going larger especially with distributed encoding is inevitably your slowest server or both chunks of a server running 2 encoders will end up being the last couple chunks to finish up. Add that to the last chunk sometimes being close to double it's size. An encoding job could end up being 20-30 minutes longer depending on how those last couple of chunks line up.

Yes, there's that problem, and that has been debated a bit over the years, that why the last chunk had to be bigger, instead of possibly very small.

But one possible advantage of less chunks is the "waiting time" between chunks, sometimes it can take a reasonable amount of time for the processing to start.

And that brings up another variable in all of this is total time for an encoding job. Just looking a FPS is only half the story. One could look at 2 similar length jobs and 1 could be significantly faster running 2 encoding windows per server, and the other could be significantly faster running just 1 encoding window per server depending on how things line up and there is no way to tell ahead of time. There is no way to really get around this, it is just the nature of the beast. Eventually I guess it would all even out.

Absolutely, you don't know what you're in for, when EVERY job.

Don't get me wrong, I am really pleased with the performance of the 59 and 79 50's once I get them setup properly. Right now this whole thing of why we have to knock down the 16 core Ryzens is more of an itch I keep scratching. Even though if we do eventually figure out what is going on, I doubt the performance increase would be all that much more than what we are getting now. It's more a curiosity now.

Unfortunately my 5950X is down, as I used some "important" items to build the 13900KF, and it won't be back online 'til at least mid to late February , but I do have another "little" 16 core, in the 3950X.

I wonder if there are many users out there that have Threadripper's doing RB, they would have to have interesting problems with 64 cores etc....

[Boulder]

Originally Posted by rlev11

Tried earlier with SetMemoryMax(8192) along with the cache setting and saw no difference. My only question with that is where would I see the memory being used by avisynth.

If ffmpeg is the one doing the frameserving (and possibly encoding too), its memory consumption is the one to look at. I always use the Details tab in Task Manager to see things better.

Originally Posted by Boulder

If ffmpeg is the one doing the frameserving (and possibly encoding too), its memory consumption is the one to look at. I always use the Details tab in Task Manager to see things better.

Hi Boulder, thanks for your help...

I just checked 2 of my PC's, (that were currently encoding) and they are using between 5,200,000 & 5,450,000 K.

[Ryushin]

It's my theory that the avisynth threads is waiting on frames when using SMDegrain/MDegrain. During my testing I watched the CPU performance during each increase in threading of avisynth threads. As long as I left a bit of headroom for the CPU to bounce between 85-100% usage everything worked great. But as soon as I increased the avisynth threads to high, the CPU went to and stayed at a 100%, everything drastically slowed down. And yea, this seem more pertaining to 4K content and not 1080p as matching 16 avisynth threads to 16 x264/x265 with 1080p was humming right along. So it's a bit of a balancing act to feed two different processes that can process data at different rates that rely on each other.

Originally Posted by Ryushin

It's my theory that the avisynth threads is waiting on frames when using SMDegrain/MDegrain. During my testing I watched the CPU performance during each increase in threading of avisynth threads. As long as I left a bit of headroom for the CPU to bounce between 85-100% usage everything worked great. But as soon as I increased the avisynth threads to high, the CPU went to and stayed at a 100%, everything drastically slowed down. And yea, this seem more pertaining to 4K content and not 1080p as matching 16 avisynth threads to 16 x264/x265 with 1080p was humming right along. So it's a bit of a balancing act to feed two different processes that can process data at different rates that rely on each other.

Glad you've joined the discussion, again

So, interesting you say that, I did a run this afternoon, upscaling a 1080p to 4K, using a deep_size script, as well as a tr=4 strength filter, and I also changed it to 3 minute chunks, (the last chunk was only about 800 frames bigger) and ONLY 1 DE server per PC, had 4 PC's going, and the speeds were really good, and like you said, with the CPU usage bouncing between 80% & 100% or so, things were running sweet

The 7950X & the 13900KF were swapping fastest FPS's all the way thru, they are SO, SO close to each other.

I have changed my encoder command lines a fair bit, they are a little different on each PC, and for now, seems to be working the best.

[Atak_Snajpera]

My last idea. Have you tried changing process priority once encoding is running for ffmpeg.exe from default to for example HIGH. x265.exe keep as is. You can use Process Hacker/Explorer for that.

Originally Posted by Atak_Snajpera

My last idea. Have you tried changing process priority once encoding is running for ffmpeg.exe from default to for example HIGH. x265.exe keep as is. You can use Process Hacker/Explorer for that.

I haven't, but wouldn't you have to change this every time you start RB ??

What about the encoder "priority" setting in RB, would that make any difference ??

Also, Process Hacker has been superceded...

https://www.systeminformer.com/

UPDATE:- You CAN set this to a HIGH priority with System Informer (Process Hacker), but to set it permanently, you need to have the app set up as Administrator (install it with Admin rights)

[apol847]

Originally Posted by Atak_Snajpera

Sorry for delay. I completely forgot about that.
https://www.mediafire.com/file/r2hmt...Source.7z/file

No problem. I was also busy. Thank you. Let's see if I get this converted to Linux.

Atak, I have a simple request that will make it a lot easier for us "old fart's" that have poor eyesight, and can't be bothered wearing their glasses.

Could you please change the colour of the text in most windows (Job Queue, Encoder Server, etc) from what seems to be a grey / dark grey, to just plain black, especially the Server name in Encoder Client.