RipBot264 v1.18.3 - Simple and easy to use GUI -> IPOD . PSP . CONSOLES . BLURAY

Originally Posted by rlev11

Just not sure where to set frames into the ripbot script or the proper syntax to use in the job script. Don't see a "/avisynth-prefetch-frames" setting in the help for the encoding server command line like threads. If it actually the frames and not threads that's killing it, once we figure out where and how to set the frames, we could set threads to 16 (or total physical cores), and just play with the threads to see where things go south and see if that same number works across different platforms. It may just end up that 1 proper frames count may be optimized for BOTH 4k and 1080p which would be perfect, or at the minimum very little difference between them. Set the frames once and be done with it.

Ideal test would be to set threads to logical cpu's, then find happy frame count doing a full frame 4k. The run 1080p and start increasing the frames and see if there is any appreciable performance gain.

That command line may not be where the threads setting needs to go, maybe it's a script thing ?!?!?

If it IS a command line option, Atak may need to change something to allow such a setting....

A buffer size increase might work as well, but could cause issues using distributed if you have servers with varying memory sizes. I would think the buffer size would get sent out to all the servers and if you have one with say 16gig instead of a setting optimized for 32 or 64, things could go crazy real fast.

Setting it to what Boulder suggested could max out any PC that has equal to, or less than 32Gb, but he did mention he hasn't seen it much higher than 23 - 24Gb, so I guess that needs to be observed, and if you have some with only 16Gb, you might have to add some.

Just a quick initial test, On my 1080p test with Mdegrain3-200, I set avisynth prefetch threads to 32 on my 7950x, so 64 prefetch frames, and I saw about a 9% speed increase just on that alone.

So I'm curious why you're using MDegrain for you tests ??

I hope you keeping an eye on this setting on the Settings page:-

"Limit to following filters only"....MDegrain needs that checked, SMDegrain doesn't !!!!!

I think you should be concentrating your tests using SMDegrain, tho.

[LigH]

Atak_Snajpera, Pauly Dunne, whoever keeps developing RipBot264:

Before the patches to support MVC were committed, specifying the input file name for x265 was optional.

Now that x265 may be able to read multiple input sources, it may also be mandatory now to specify them with a preceding parameter "--input".

[rlev11]

Originally Posted by FTLOY

So I'm curious why you're using MDegrain for you tests ??

I hope you keeping an eye on this setting on the Settings page:-

"Limit to following filters only"....MDegrain needs that checked, SMDegrain doesn't !!!!!

I think you should be concentrating your tests using SMDegrain, tho.

Just using mdegrain as it is what is built into Ripbot proper and also has the most profound affect of the issue with the prefetch threads/frames.

If we can figure out what the proper settings to use on high core cpu's so the built in Mdegrain works properly, should fix everything else using whatever avisynth scripts are running

Originally Posted by rlev11

Just using mdegrain as it is what is built into Ripbot proper and also has the most profound affect of the issue with the prefetch threads/frames.

If we can figure out what the proper settings to use on high core cpu's so the built in Mdegrain works properly, should fix everything else using whatever avisynth scripts are running

Fair enough....I did some quick tests with a simple SMD script, altering the commandline, unchecking those boxes, adding a extra command to a script, and I got a lot of mixed results, some good, some bad, but I sort of lost track, so I will have to go over things again, and try and jot it down as I go.

Now there seems to be some major change to x265

Also, more & more info about 9950X's, too.

[Atak_Snajpera]

If I had 16c cpu I would do this:
1) create 4k job in standard mode (no de)
2) manually open job1.avs in notepad and then add those at the very beginning

SetCacheMode(CACHE_OPTIMAL_SIZE)

or / and

SetMemoryMax(32768)

3) And then play with

Prefetch(16,16) #(threads,frames)

4) encode manually via

job1_EncodeVideoPass1.CMD

file

or just check how fast job1.avs is being processed in AVSMeter.

Originally Posted by Atak_Snajpera

If I had 16c cpu I would do this:
1) create 4k job in standard mode (no de)
2) manually open job1.avs in notepad and then add those at the very beginning

SetCacheMode(CACHE_OPTIMAL_SIZE)

or / and

SetMemoryMax(32768)

3) And then play with

Prefetch(16,16) #(threads,frames)

4) encode manually via

job1_EncodeVideoPass1.CMD

file

or just check how fast job1.avs is being processed in AviSynth Meter.

But what if this was added to a custom script ??

How would you add the Prefetch(16,16) to a script to work with RB ??

Your above suggestion, does that make DE redundant ??

======================

Have your read anything about the changes to x265 ??

[rlev11]

Originally Posted by Atak_Snajpera

If I had 16c cpu I would do this:
1) create 4k job in standard mode (no de)
2) manually open job1.avs in notepad and then add those at the very beginning

SetCacheMode(CACHE_OPTIMAL_SIZE)

or / and

SetMemoryMax(32768)

3) And then play with

Prefetch(16,16) #(threads,frames)

4) encode manually via

job1_EncodeVideoPass1.CMD

file

or just check how fast job1.avs is being processed in AVSMeter.

just setting
SetCacheMode(CACHE_OPTIMAL_SIZE)
SetMemoryMax(16384)

running GOT mdegrain3-200 resulted in a 300% increase from little over 2fps to 6.4 fps which is right around setting prefetch treads to 12 in previous tests

tried 32 and 64 gig in that memory setting and saw very little difference. In my task manager adding in the memory setting ffmpeg.exe memory usage jumped from around 7 gig to 10-11 gig regardless of what I put in for memory cache size

so this is what is in job1.avs
#Prefetch
video=Prefetch(video,16)

so how would I change this syntax for the Prefetch(16,16) #(threads,frames) setting

We are making progress....

[rlev11]

And let me just add, step 1 now is just trying to figure out exactly where the bottleneck is occurring. Just figuring this out is success in my mind. Is it prefetch threads, frames, memory allocation, or some hybrid combination of them all?? Once we figure this out, then the next step is going to see how those translate to other high core processors. Right now I am testing with my 7950x. Remains to be seen if the same settings will work on say a 5950x,3950x, or a high core Intel, although they probably will.

After that will come how these results can best be integrated into Ripbot that are both easy to set, and a novice user will understand. There is a point where too many settings can overload and frustrate users. We also have distributed encoding to add into the mix and how that will play with any settings we deem as wanting to set.

[Boulder]

It's clear that the Avisynth default cache (512MB or something like that..) is not enough for your use case.

[rlev11]

Originally Posted by Boulder

It's clear that the Avisynth default cache (512MB or something like that..) is not enough for your use case.

Looks like AviSynth+ default is 4GB according to the wiki. 1080p does not come close to that, I saw about 2 gig for ffmpeg during a 1080 run,

Thanks....

[Atak_Snajpera]

Originally Posted by rlev11

Looks like AviSynth+ default is 4GB according to the wiki. 1080p does not come close to that, I saw about 2 gig for ffmpeg during a 1080 run,

Thanks....

What is the lowest memory value without performance drop? 8192?

[Boulder]

Originally Posted by rlev11

Looks like AviSynth+ default is 4GB according to the wiki. 1080p does not come close to that, I saw about 2 gig for ffmpeg during a 1080 run,

Thanks....

"DefaultMemoryMax = minimum(physical_memory / 4, secondary_memory_max_limit)" with maximum limit at that 4GB.

I'm just a bit confused if it's actual installed physical memory, or free memory as mentioned in the wiki.

8GB would probably match most cases.

[rlev11]

Originally Posted by Atak_Snajpera

What is the lowest memory value without performance drop? 8192?

I will let you know tomorrow

Originally Posted by rlev11

I will let you know tomorrow

I've been fiddling around with this all afternoon, and not too sure if I have it going any better, or not, but it's easy to make it worse....

The memory value doesn't have a huge difference, but I would suggest probably not going below 16384...

But one thing if this call SetCacheMode(CACHE_OPTIMAL_SIZE) is added to a script, it's not too good, I've found that SetCacheMode() seems the best.

SetCacheMode(mode)  
AVS+Fine tunes the internal frame caching strategy in AviSynth+.
Available values:
0 or CACHE_FAST_START start up time and size balanced mode (default)
1 or CACHE_OPTIMAL_SIZE slow start up but optimal speed and cache size
since Avisynth+ 3.6.1

And video=Prefetch(video,32,16) doesn't do a lot, either

Atak,

I would like to make you aware that apparently upcoming x265 builds have been altered/updated, call it what you will, but app's that use it may have to edit the way that x265 commands are written...

I have been chatting with LigH, and he has been kind enough to show what needs to be done, so please go here, and have a little read:-

https://forum.doom9.org/showthread.p...50#post2005750

And before you say that I shouldn't be using modded builds, even the latest "vanilla" builds are having the same "problem".

If you download this build and try it for yourself, then you might be able to figure out what needs to be done with RB.

https://forum.doom9.org/showthread.p...47#post2005647

[LigH]

I did not compile a "modded" build. Only a very recent one. Multicoreware changed the rules for the CLI parsing because they need it to support multiple input streams for MVC.

[rlev11]

Originally Posted by Atak_Snajpera

What is the lowest memory value without performance drop? 8192?

Ran the same tests with GOT with same Mdegrain3-200, running the job1_EncodeVideoPass1.cmd manully.

First thing I found is SetCacheMode(CACHE_OPTIMAL_SIZE) make little if any difference. In fact SetCacheMode() appears to a a couple percent faster.

Ran the same script changing the SetMemoryMax(xxxxx) number and came up with the following numbers which I recorded when the job went to 2% on each run.

4096 - 2.28fps
5120 - 4.52fps
6144 - 6.15fps
8192 - 6.50fps
10240 - 6.50fps
16384 - 6.51fps

So looks like 8192 at least in this case is the magic number. Is similar to what I see when changing prefetch threads to 12. I still have not done these same tests on my 3950x, 5950x, or i7-14700 to make sure I see similar results but it would make sense if it did.

Atak, I can do any additional testing you want with my variety of high core cpu's. I understand the complexity of taking a small testing sample and thinking about how best to incorporate anything into a program update. If you would rather take this offline for now, just PM me and we can setup anything you want as far as testing.

Additional testing has showed setting the 3950x and 5950x to prefetch=16 and the 14700 to prefetch=20 with the SetMemoryMax(8192) shows the same improvement as the 7950x

[rlev11]

So it is looking like this effort is having some positive results for me now. End result is now we no longer have to cripple the thread count of the high core cpu's down to 12-14 when doing 4k encoding regardless of using the built in mdegrain or in my case using a custom smdegrain script.

The bottleneck for 4k encoding is the avisynth max memory setting. For both mdegrain and my custom smdegrain scripts having SetMemoryMax(8192) restores performance on the high core cpu's in my results I am seeing. I tried 16384 and saw very little difference (1-3%) so no need to go above 8192 unless you need to for other reasons.

So now I will set prefetch threads to number of physical cores regardless on all servers, add in SetCacheMode() and SetMemoryMax(8192) to all my custom degraining scripts, and enjoy a little better performance over my previous method of crippling my R9's to 12 prefetch threads. Back to using my full 13 server distributed encoding farm, On my same GOT test I have been using, using SetMemoryMax(8192) was about 7% faster than prefetch=12 in total fps. This should also speed up 1080p and below a bit since I don't have to remember to switch the high core encoding servers to full prefetch (16 or 20 depending on the cpu)

Not sure how this may translate to other users workflows though, guess time will tell... but thanks for suggestions and input on this issue that has been bugging me for years now

Originally Posted by rlev11

So it is looking like this effort is having some positive results for me now. End result is now we no longer have to cripple the thread count of the high core cpu's down to 12-14 when doing 4k encoding regardless of using the built in mdegrain or in my case using a custom smdegrain script.

I still think you should be using SMD scripts for your tests, as you so rarely use MD, and the results could be quite different !!!

The bottleneck for 4k encoding is the avisynth max memory setting. For both mdegrain and my custom smdegrain scripts having SetMemoryMax(8192) restores performance on the high core cpu's in my results I am seeing. I tried 16384 and saw very little difference (1-3%) so no need to go above 8192 unless you need to for other reasons.

Also interesting that you've "settled" for only 8192, when (in your previous post), 16384 was a smidgen faster.....I tested up to 32768, and even tho there was only a very slight difference between the three, 32 was faster.

So now I will set prefetch threads to number of physical cores regardless on all servers, add in SetCacheMode() and SetMemoryMax(8192) to all my custom degraining scripts, and enjoy a little better performance over my previous method of crippling my R9's to 12 prefetch threads. Back to using my full 13 server distributed encoding farm, On my same GOT test I have been using, using SetMemoryMax(8192) was about 7% faster than prefetch=12 in total fps. This should also speed up 1080p and below a bit since I don't have to remember to switch the high core encoding servers to full prefetch (16 or 20 depending on the cpu)

Your comment (also in the previous post) regarding SetCacheMode(CACHE_OPTIMAL_SIZE), I totally agree, it slowed things down, for me, and SetCacheMode() was the best option

There was no mention of the Prefetch threads/frames setting ?? did you try it ??

I have also tweaked several other settings that dramatically change the speed, but the proof will be when encoding full length movies, etc, not just small test samples

Not sure how this may translate to other users workflows though, guess time will tell... but thanks for suggestions and input on this issue that has been bugging me for years now

Exactly, depends so much on hardware setup, scripts used....lot's of variables.

My testing will continue for a little while yet, just to be sure.

The next obstacle, is the new generation of x265...that WILL need to be addressed, soon.

[Juha]

Is it anyway possible to customize audio options? I have source material that has PCM and DTS audio tracks. They both have 2 channels. I'd like to convert them to AAC and use 256 Kbps at least but Ripbot doesn't allow to use higher bitrate for 2 channel AAC audio than 128 Kbps.

I tried to mess around in the program settings but despite what changes I make there, I get no different options.