Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 Encoder GUIs

Reply
 
Thread Tools Search this Thread Display Modes
Old 30th May 2022, 20:41   #19501  |  Link
slalom
Registered User
 
slalom's Avatar
 
Join Date: Jan 2010
Posts: 456
You don't have to, you setup everything in DE settings
__________________
E5 2697 v2 @ 3.0GHz on P9X79 Deluxe 24GB
Xeon E5-2680 v2 @ 3.1GHz 16GB
Sony Vaio VPC-F13Z1E/B
slalom is offline   Reply With Quote
Old 31st May 2022, 04:46   #19502  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 965
Quote:
Originally Posted by slalom View Post
You don't have to, you setup everything in DE settings
Yes, he's got it working now, I PM'ed him

It's just that there's no relevant instructions on how to do it, anymore.
__________________
Long term RipBot264 user.

RipBot264 modded builds..
TDS is offline   Reply With Quote
Old 7th June 2022, 03:14   #19503  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 692
Time for another update...

Hi,

There are several significant updates in this build (read the changelog)

Also, I have completely re-organised the Script & Custom Scripts folders, so it's not so cluttered in the Custom Script list.

https://forum.doom9.org/showthread.p...72#post1970572

And as always, backup your existing Tools folder before using this, just to be sure.

Good luck.
__________________
Not poorly done, just doin' it my way !!!
Live every day like it's your last, because one day, it will be !! (M$B)

Last edited by Pauly Dunne; 20th June 2022 at 04:27.
Pauly Dunne is offline   Reply With Quote
Old 8th June 2022, 00:49   #19504  |  Link
MCFish
Registered User
 
MCFish's Avatar
 
Join Date: Oct 2002
Location: Norway
Posts: 46
Thanks man
__________________
MCFish
MCFish is offline   Reply With Quote
Old 9th June 2022, 04:38   #19505  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 965
I'm curious why my Ryzen 3950X & 5950X (16 core) are encoding quite a bit slower than the 5900X (12 core), even an old dual Xeon X5690 is faster.

This a 4K encode, but on other encodes they "catch up".

https://i.imgur.com/xRW77Rb.jpg
__________________
Long term RipBot264 user.

RipBot264 modded builds..
TDS is offline   Reply With Quote
Old 9th June 2022, 13:55   #19506  |  Link
Ryushin
Registered User
 
Ryushin's Avatar
 
Join Date: Mar 2011
Posts: 431
Quote:
Originally Posted by TDS View Post
I'm curious why my Ryzen 3950X & 5950X (16 core) are encoding quite a bit slower than the 5900X (12 core), even an old dual Xeon X5690 is faster.

This a 4K encode, but on other encodes they "catch up".

https://i.imgur.com/xRW77Rb.jpg
Are you degraining? Did you uncheck that box to "Limit to the following filters only"? If you are using Pauly Dunne's Tools, make sure you add SMDegrain to the list. Use Atak's experience here. Not every one of the pieces in RB needs to be threaded and only certain processes can take advantage of it and too many threads can slow down the machine.

In my post a few weeks ago, when I got my new to me dual CPUs with 14 cores and 28 threads I had awful performance when using MDegrain at the time. Once I dug into it I found I had to limit the number threads.
https://forum.doom9.org/showthread.p...47#post1968247
In my case, for that system, 14 threads was optimal for my general encoding even though I had 48 CPU/Threads.

My suggestion is to use a grainy 4K 15-20 minute sample as your test and tune each encoding server individually with degraining enabled. Enable the "Limit to the following filters only" option. Change the chunk size to 2-5 minutes (for testing), enable only one Server process. Then tune that machine to the number of threads that get the highest fps possible. Watch the Task Manager during this time and if you are getting less than 80% utilization and if so, after tuning for one Server process, increase it to two and see what your overall FPS is. Once each computer has been tuned for the number of threads, then run your encodes again.

Your set up should just fly when each machine is tuned.
Ryushin is offline   Reply With Quote
Old 9th June 2022, 14:50   #19507  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 965
Quote:
Originally Posted by Ryushin View Post
Are you degraining? Did you uncheck that box to "Limit to the following filters only"? If you are using Pauly Dunne's Tools, make sure you add SMDegrain to the list. Use Atak's experience here. Not every one of the pieces in RB needs to be threaded and only certain processes can take advantage of it and too many threads can slow down the machine.

In my post a few weeks ago, when I got my new to me dual CPUs with 14 cores and 28 threads I had awful performance when using MDegrain at the time. Once I dug into it I found I had to limit the number threads.
https://forum.doom9.org/showthread.p...47#post1968247
In my case, for that system, 14 threads was optimal for my general encoding even though I had 48 CPU/Threads.

My suggestion is to use a grainy 4K 15-20 minute sample as your test and tune each encoding server individually with degraining enabled. Enable the "Limit to the following filters only" option. Change the chunk size to 2-5 minutes (for testing), enable only one Server process. Then tune that machine to the number of threads that get the highest fps possible. Watch the Task Manager during this time and if you are getting less than 80% utilization and if so, after tuning for one Server process, increase it to two and see what your overall FPS is. Once each computer has been tuned for the number of threads, then run your encodes again.

Your set up should just fly when each machine is tuned.
Yes, I was degraining a grainy 4K, and I had "Limit to the following filters only" unchecked, and threads at "0".

Then I did add SMDegrain to the list, and checked the "Limit to the following filters only", and changed the Threads to 24, didn't make a noticable difference, if anything, it was a little slower.

But now that you have provided a fairly detailed process of "tuning", I will definitely take some time to get it set up.

Having said that, each encode can be quite different to another, even each chunk can change, due to it's "content's".

Well done, thanks.

Edit:-

Well after spending a couple of hours testing this process, with & without SMDegrain in the "list", trying many different threads, and I thought I'd found the sweet spot @ 22 threads.

And I found the difference between having SMDegrain in the "list", and "Limit to the following filters only" checked or not, made very little difference.

So at this point, I'll just have to put up with the 16 Core Ryzens being slower than the 12 Core 5900X (at least on this particular encode), I'm using a pretty high tr setting.

video=SMDegrain(video,tr=10,thSAD=1000,thSADC=500,thSCD1=400,thSCD2=130,contrasharp=true,prefilter=2,str=1.2,refinemotion=true)
__________________
Long term RipBot264 user.

RipBot264 modded builds..

Last edited by TDS; 10th June 2022 at 05:39.
TDS is offline   Reply With Quote
Old 10th June 2022, 04:29   #19508  |  Link
wamar21
Registered User
 
Join Date: Feb 2022
Posts: 2
Quote:
Originally Posted by TDS View Post
Yes, he's got it working now, I PM'ed him

It's just that there's no relevant instructions on how to do it, anymore.
Here's a quick setup on a fresh win10 install

https://youtu.be/tftERYq94GE

Last edited by wamar21; 11th June 2022 at 12:03. Reason: hyperlink
wamar21 is offline   Reply With Quote
Old 10th June 2022, 05:30   #19509  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 965
Quote:
Originally Posted by wamar21 View Post
Here's a quick setup on a fresh win10 install

https://youtu.be/A3tWMGkmcaM
Best viewed in 1080p, turn the sound down, and play at 0.5 speed.

Welcome to the Forum, and I'm sure a LOT of people will appreciate that you took the time to do this.

Even tho you're using a VM, the basic steps are the same.

Cheers
__________________
Long term RipBot264 user.

RipBot264 modded builds..

Last edited by TDS; 10th June 2022 at 08:09.
TDS is offline   Reply With Quote
Old 10th June 2022, 20:12   #19510  |  Link
wamar21
Registered User
 
Join Date: Feb 2022
Posts: 2
Thanks, I also noticed a few ambiguities I wanna clear up anyway. Hopefully this one stays useful for as long as possible. Ill tweak it a bit.

I wanted it to be somewhere between "good, clear, useful tutorial" and "annoying 2000s homage/shitpost" but I may have rode that line a bit too close.


UPDATE: Removed the "brb helping w/ groceries" bit for clarity's sake, RIP cultural artifact.

Last edited by wamar21; 11th June 2022 at 12:06.
wamar21 is offline   Reply With Quote
Old 11th June 2022, 13:22   #19511  |  Link
Ryushin
Registered User
 
Ryushin's Avatar
 
Join Date: Mar 2011
Posts: 431
Quote:
Originally Posted by TDS View Post
Well after spending a couple of hours testing this process, with & without SMDegrain in the "list", trying many different threads, and I thought I'd found the sweet spot @ 22 threads.

And I found the difference between having SMDegrain in the "list", and "Limit to the following filters only" checked or not, made very little difference.

So at this point, I'll just have to put up with the 16 Core Ryzens being slower than the 12 Core 5900X (at least on this particular encode), I'm using a pretty high tr setting.

video=SMDegrain(video,tr=10,thSAD=1000,thSADC=500,thSCD1=400,thSCD2=130,contrasharp=true,prefilter=2,str=1.2,refinemotion=true)
So each of your encoding servers is set for 22 threads? I would think each one would be different based on number of cores, IPC capabilities, etc. I would probably actually run just MDegrain2 for the tests to get your baseline. I know there is some variance in encoding speed when dealing with different aspect ratios and 4K vs HD.
Ryushin is offline   Reply With Quote
Old 11th June 2022, 14:48   #19512  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 965
Quote:
Originally Posted by Ryushin View Post
So each of your encoding servers is set for 22 threads? I would think each one would be different based on number of cores, IPC capabilities, etc. I would probably actually run just MDegrain2 for the tests to get your baseline. I know there is some variance in encoding speed when dealing with different aspect ratios and 4K vs HD.
No, they are not...I was only testing on the Client PC, which is one of the 16 Core Ryzen's, but I ended up just putting it back to "0" for the Threads, Unchecked "Limit..." option, and removed SMDegrain from the list.

It seems that it's the particular filter I am using, a VERY strong high "tr" #.

When I go back to using "normal" filter's, ALL the Ryzen's are pretty much equal.

Fortunately, I don't think I have too many "extra" noisy 4K's to filter, and it's not much of an issues with lesser resolutions.

I might give MDegrain2 a go when I have some spare time, but for now I just want to get some encoding done, while the weather is nice & sunny, which it hasn't been for months & months.

As I don't "waste" electricity doing encodes on overcast days, or over night !!! It's a fine weather, Solar Energy only, process !!

EDIT:-

SSDD....so I tried something a little weird today, I actually limited the Cores available to the CPU in BIOS, so the 16 core CPU, would now run with 12 cores !!!

Surprisingly enough, it made a significant difference...FASTER !!!, but only on that very heavy filter script.

Another thing I tried was the x265 commands...I added --pools 24 that didn't seem to have much of an affect.
__________________
Long term RipBot264 user.

RipBot264 modded builds..

Last edited by TDS; 12th June 2022 at 06:09.
TDS is offline   Reply With Quote
Old 12th June 2022, 11:25   #19513  |  Link
Ryushin
Registered User
 
Ryushin's Avatar
 
Join Date: Mar 2011
Posts: 431
Quote:
Originally Posted by TDS View Post
No, they are not...I was only testing on the Client PC, which is one of the 16 Core Ryzen's, but I ended up just putting it back to "0" for the Threads, Unchecked "Limit..." option, and removed SMDegrain from the list.

SSDD....so I tried something a little weird today, I actually limited the Cores available to the CPU in BIOS, so the 16 core CPU, would now run with 12 cores !!!

Surprisingly enough, it made a significant difference...FASTER !!
This is why you have to tune the CPU threads for each of your encoding Servers (per computer). So run a 1 minute chunk and time each one just using MDegrain2. Adjust the threads until you find the sweet spot for the encoding server. I would still use the Limit to the following filters option enabled though.

I think if you spend a couple of hours or so tuning your threads for each of your Servers it will pay big dividends in encoding time vs watts used.

Atak: You might want to create a tool that uses MDegrain2/SMDegrain that will find the sweet spot for thread use per Computer. As CPU core count seems to be increasing more and more people are going to hit this issue.
Ryushin is offline   Reply With Quote
Old 12th June 2022, 14:31   #19514  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 965
Quote:
Originally Posted by Ryushin View Post
This is why you have to tune the CPU threads for each of your encoding Servers (per computer). So run a 1 minute chunk and time each one just using MDegrain2. Adjust the threads until you find the sweet spot for the encoding server. I would still use the Limit to the following filters option enabled though.

I think if you spend a couple of hours or so tuning your threads for each of your Servers it will pay big dividends in encoding time vs watts used.

Atak: You might want to create a tool that uses MDegrain2/SMDegrain that will find the sweet spot for thread use per Computer. As CPU core count seems to be increasing more and more people are going to hit this issue.
I will try this other procedure when I get a chance, but this "problem" only seems to be A "problem" when using very heavy filters, other wise, it's working great, and CPU utilization is basically 100%.

Even when it's encoding very slowly, it's still pushing the CPU's to pretty much 100%, can't ask anymore than that.

Correct me if I'm wrong, BUT all the encoding servers use the RipBot settings of the client machine, so that individual server settings "do not apply" !!!

As Atak mentioned once, in reality the server PC's ONLY need the Encoding Server, not the whole "app", you'd only need to manually start the Encoding Server, and DE will do the rest, so there's no settings required.

But it IS strange that in some situations, a 12 core system, does a faster job than a 16 core system....
__________________
Long term RipBot264 user.

RipBot264 modded builds..

Last edited by TDS; 12th June 2022 at 14:40.
TDS is offline   Reply With Quote
Old 12th June 2022, 15:27   #19515  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
You can manually define number of prefetch threads for each server like this
Code:
EncodingServer.exe /avisynth-prefetch-threads 8
By the default prefetch threads = number of physical cores on current PC

Last edited by Atak_Snajpera; 12th June 2022 at 15:30.
Atak_Snajpera is offline   Reply With Quote
Old 13th June 2022, 02:00   #19516  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 965
Quote:
Originally Posted by Atak_Snajpera View Post
You can manually define number of prefetch threads for each server like this
Code:
EncodingServer.exe /avisynth-prefetch-threads 8
By the default prefetch threads = number of physical cores on current PC
Thanks for chiming in, Atak...

You know, that option has been there all along, staring us in the face, but not really knowing what it does.

So, under "Settings", "Distributed Encoding" tab, click on the little "Spanner", then "Command line help".....

EDIT:-

I just tried this on my 3950X 16 Core (which is currently running as an encoding server), and adding that command, only using 24 instead of 8, hasn't changed anything, that I can detect.

In the Encoding Server "window", it still shows :- x265 [info]: Thread pool created using 32 threads

So again, ALL the server machines take their "instructions" and "settings" from the main Client PC, regardless how the server PC is setup !


EDIT #2:-

OK, changed the command line on the Client PC, and even tho it states that Avisynth is only using 24 threads, the x265 info remains @ what ever the CPU provides.

The ONLY way it seems, to change the amount of threads that x265 uses is in the command line for x265, by adding --pools [integer]
__________________
Long term RipBot264 user.

RipBot264 modded builds..

Last edited by TDS; 13th June 2022 at 03:05.
TDS is offline   Reply With Quote
Old 13th June 2022, 12:20   #19517  |  Link
Ryushin
Registered User
 
Ryushin's Avatar
 
Join Date: Mar 2011
Posts: 431
Quote:
Originally Posted by TDS View Post
Even when it's encoding very slowly, it's still pushing the CPU's to pretty much 100%, can't ask anymore than that.
But it IS strange that in some situations, a 12 core system, does a faster job than a 16 core system....
Yea, I saw a 100% CPU as well when I had unlimited threads. A bottleneck is still taking place. By optimizing the threads, you are removing the bottleneck that was causing slower processing.
Ryushin is offline   Reply With Quote
Old 13th June 2022, 16:43   #19518  |  Link
Ryushin
Registered User
 
Ryushin's Avatar
 
Join Date: Mar 2011
Posts: 431
RipBot 2.0

Hi Atak,

If your schedule allows, it would be nice to get a RipBot 2.0. You might want to change the name to just RipBot or maybe something like RipBotEncoder.

SMDegrain has been a game changer in quality, though not without its steeper learning curve. I have transitioned over to SMDegrain exclusively though using Pauly Dunnes Tools. Though I create my own scripts to use the CPU only instead of the GPU. SMDegrain is powerful and flexible enough to be the primary degrainer for RB. The options for Blur plus Contrasharp on very grainy sources produces output that defies what I would think would be possible.

Dolby Vision is also easier to deal with and is scriptable: https://forum.doom9.org/showthread.php?t=183479 There are some options that have to be passed if cropping is going to be used in the final output.

It would be nice to see a UI overhaul of the AVI Synth settings. If the window could be made quite a bit larger to see the 2.40:1 bottom borders for 4K sources like you can with HD sources. The larger window will also give you more space for the options. For the SMDegrain settings. A pull down menu with a few presets for most people, something like this:
Light: SMDegrain(video,tr=2,thSAD=200,thSADC=100,refinemotion=true)
Medium: SMDegrain(video,tr=4,thSAD=400,thSADC=200,refinemotion=true)
Heavy: SMDegrain(video,tr=8,thSAD=800,thSADC=400,refinemotion=true)
Then add a Blur+Sharp radio button next to SMDegrain filter that adds the following depending how heavy they wanted it:
Light: contrasharp=true,prefilter=0
Medium: contrasharp=true,prefilter=1
Heavy: contrasharp=true,prefilter=2
Then having an Advanced option on the bottom of the pull down that opens up another new window with more switches and options. Such as having the tr, thSAD (I think thSADC should always be half of thSAD), refinemotion, contrasharp, prefilter, str (for darker sources), blksize, etc. A little "?" mark next to each option would be nice with a mouse hover pop up that provides details on that option.

Changes for the Distributed Encoding model would be good too. The Primary server would dole out work to the clients. The RipBot install on the client would have something like a "DE Client" startup option and then RB will just start up minimized to the system tray with no Server processes started. The Primary RB server would then communicate with the RB Client to start up how many server processes along with the options for those server processes. The DE window can also have a per Client Thread Calibration button that you can click that will open up a window with two grainy 1 minutes sources, one for HD and one for 4K that the user can choose select one they want, then pop up a warning that this could take a long time, then the Primary server will send the Job to the RB Client, start up a single Server Process along with the number of threads to test for SMDegrain Heavy or MDegrain2-400, over and over until the optimum number of threads is found. Though some intelligence can be added, like start with half threads, then the lower quarter threads, upper quarter of threads, then fine tune it between the two quarters that had the best speed.

Atak, I know I've asked this before, but it would be nice to get RB up on Git along with the source and a issue tracker.
Ryushin is offline   Reply With Quote
Old 14th June 2022, 04:06   #19519  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 965
Quote:
Originally Posted by Ryushin View Post
Yea, I saw a 100% CPU as well when I had unlimited threads. A bottleneck is still taking place. By optimizing the threads, you are removing the bottleneck that was causing slower processing.
Hi, are you reading everything I've said ??

It's ONLY happening with the VERY very strong filter I am using, on ALL the other filters, they are all pretty much equal encoding speed.

So there must be something going on when you bump up the "tr", "thSAD" & "thSADC" levels/numbers.

Up to "tr=8" is fine...

But one thing I need to repeat, is, ALL the server PC's, regardless of any tweaks & tuning, take ALL their settings & instructions from the client PC.

So if you went to all the trouble of tuning every server, when running them as a client, is all over ridden, when they are used as a server, because the servers ONLY require the Encoding Server to do the DE process.

Now I'm pretty sure that running RipBot on a VM, wouldn't change the fundamentals, and I don't use VM's for ANYTHING !!

But there's obviously a significant difference between your hardware & setup, to what I have.

I have tried everything that has been suggested, and have not seen any significant change, so I think we need to put this topic to rest, it's just going around in circles.
__________________
Long term RipBot264 user.

RipBot264 modded builds..
TDS is offline   Reply With Quote
Old 14th June 2022, 04:48   #19520  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 692
Quote:
Originally Posted by Ryushin View Post

SMDegrain has been a game changer in quality, though not without its steeper learning curve. I have transitioned over to SMDegrain exclusively though using Pauly Dunnes Tools. Though I create my own scripts to use the CPU only instead of the GPU. SMDegrain is powerful and flexible enough to be the primary degrainer for RB. The options for Blur plus Contrasharp on very grainy sources produces output that defies what I would think would be possible.
WOW, I'm blown away with this post !!!

I knew when I started to implement SMDegrain into RipBot, that is was superior, and that was before I realised that there were options to use pre-filters as well.

It wasn't until I discovered Dogway's Doom9 page, that I realised all this.

https://forum.doom9.org/showthread.php?t=182881

I certainly can't take any credit for what SMDegrain can do !!!

But I have to say that there are a lot more pre-filters, than just 0,1 & 2, sure there are GPU based pre-filters that other users could benefit from.

There also many other filters available in my builds, too, not just focusing on SMDegrain.

And finally, since I first post my RipBot build(s), there has been 100's of downloads, the latest build posted only a week ago, has been downloaded over 30 times.

There has only been a few say thanks, but for Ryushin to say these words, is pretty amazing.
__________________
Not poorly done, just doin' it my way !!!
Live every day like it's your last, because one day, it will be !! (M$B)
Pauly Dunne is offline   Reply With Quote
Reply

Tags
264, 265, appletv, avchd, bluray, gui, iphone, ipod, ps3, psp, ripbot264, x264 2-pass, x264 gui, x264_64, x265, xbox360

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:43.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.