Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
![]() |
|
Thread Tools | Search this Thread | Display Modes |
![]() |
#19481 | Link |
Registered User
Join Date: Mar 2011
Posts: 349
|
SMDegrain Needs to be Added to RipBot264
So I've spent a few days getting to know SMDegrain and it's options compared to MDegrain. At first using Pauly Dunne's custom scripts for SMDegrain did not provide better results than MDegrain and were in fact worse size wise than just using MDegrain. Pauly encodes for bit rate and I prefer Constant Quality since the file size actually tells me how clean the output is.
I found good documentation for SMDegrain here: http://avisynth.nl/index.php/SMDegrain Originally I was testing with a 15 minute clip from Invasion of the Body Snatchers in 4K. But the better test case was to use Blade Runner in 4K since the 15 minutes clip from the beginning has lots of grain, haze, cigarette smoke, etc, plus it will show off color banding if there are problems. My test results: Source: Blade Runner 1982 4K - 15 Minutes cut from 2:58-17:58 Source Size: 4.0 GB Constant Quality 18 6.1 GiB: No Degraining: (Note, file size is larger than the source because of so much grain) 2.2 GiB: MDegrain2 thSAD=400 1.6 GiB: MDegrain3 thSAD=600 Note: I Find MDegrain3 can cause problems with example of a close up a face, you can see the facial texture, but if they suddenly turn their face, it blurs, then you can see the texture again when the turn finishes. Very annoying to watch and as such, I only used MDegrain3 on the absolute grainiest sources. 4.1 GiB: SMDegrain(video) (no options) (See docs for the default options) 2.8 GiB: SMDegrain(video,tr=2,thSAD=400,thSCD2=80) 2.8 GiB: SMDegrain(video,tr=2,thSAD=400,thSADC=200,thSCD2=80) 1.7 GiB: SMDegrain(video,tr=2,thSAD=800,thSADC=400,thSCD2=80) 1.9 GiB: SMDegrain(video,tr=2,thSAD=800,thSADC=400,thSCD2=80,contrasharp=true,prefilter=2,refinemotion=true) 1.0 GiB: SMDegrain(video,tr=4,thSAD=800,thSADC=400,thSCD2=80) 1.1 GiB: SMDegrain(video,tr=4,thSAD=800,thSADC=400,thSCD2=80,contrasharp=true,prefilter=2,refinemotion=true) 828 MiB: SMDegrain(video,tr=8,thSAD=800,thSADC=400,thSCD2=80) 3.1 GiB: SMDegrain(video,tr=8,thSAD=800,thSADC=400,thSCD2=80,contrasharp=true,prefilter=-1,refinemotion=true) 1.2 GiB: SMDegrain(video,tr=8,thSAD=800,thSADC=400,thSCD2=80,contrasharp=true,prefilter=0,refinemotion=true) 967 MiB: SMDegrain(video,tr=8,thSAD=800,thSADC=400,thSCD2=80,contrasharp=true,prefilter=1,refinemotion=true) 767 MiB: SMDegrain(video,tr=8,thSAD=800,thSADC=400,thSCD2=80,contrasharp=true,prefilter=2,refinemotion=true) So increasing the "tr" value is essentially which MDegrain you want to use. In the case above, from 2-8. The file size was actually quite good using a tr of 8. The contrasharp provides sharpening. I found adding the contrasharp provided a nice detail enhancement but it really needed to be paired with using a prefilter which blurs out the grain a bit before sharpening. A prefilter of -1 provides no filtering, 0 is light, 1 is medium, and 2 is strong. Blocky sources need to use 3 for the prefilter. Refinemotion helps with detecting motion, but if the prefilter is too strong it can cause problems. I watched the output of the 767 MiB file this morning and compared it to the source and the difference is outstanding. Atak, please add SMDegrain to RipBot264. It seems to be a swiss army knife of sorts. You will probably need to give perhaps three presets and hopefully and Advanced that lets you tune some of the parameters such as tr, thSAD, contrasharp, prefilter, refinemotion, etc. I almost think SMDegain could be so good you can retire MDegrain at this point. You can download the 15 minute Blade Runner source file and the 767 MiB file to see the differences from here: https://cloud.chrisdos.com/s/mdMDjgQMbMCdaes The link expires in two weeks. I've ripped some of my grainiest 4K movies to try things out on and watch them and see how it fairs. Pauly Dunne, thank you for the hard work of providing the SMDegrain scripts and tools. I'll try and take SMDegrain pieces and integrate them into to RB's default tools folder so I can turn on auto updates again. |
![]() |
![]() |
![]() |
#19482 | Link | |
Grumpy Old Man.
Join Date: Jul 2019
Location: Out There....
Posts: 643
|
Quote:
WOW, you HAVE been busy, well done !!! I am SO SO happy that you have pursued this, I knew that you would NOT be disappointed with SMDegrain, once you got your head around it. So I have compared & cross referenced some of your "scripts" with "my" filters names, and it seems that you are mainly using the E-H (extra hard) settings, and just changing the "tr" values. There are so many others to choose from with lower settings, already. For example :- 3.1 GiB: SMDegrain(video,tr=8,thSAD=800,thSADC=400,thSCD2=80,contrasharp=true,prefilter=-1,refinemotion=true) is basically the SMDegrain E-H Minblur#-1 filter, and so on... 1.2 GiB: SMDegrain(video,tr=8,thSAD=800,thSADC=400,thSCD2=80,contrasharp=true,prefilter=0,refinemotion=true) SMDegrain E-H Minblur#0 967 MiB: SMDegrain(video,tr=8,thSAD=800,thSADC=400,thSCD2=80,contrasharp=true,prefilter=1,refinemotion=true) SMDegrain E-H Minblur#1 767 MiB: SMDegrain(video,tr=8,thSAD=800,thSADC=400,thSCD2=80,contrasharp=true,prefilter=2,refinemotion=true) SMDegrain E-H Minblur#2 So if your starting from scratch with your own SMDegrain scripts, they are already there !!!, you just need to look in the Custom folder. Should I rename them to something a little more explanatory ?? Now I know you're no stranger to modifying scripts, so you can go crazy modding any of these to suit your needs, and just save them in the Custom folder, and delete any that you don't want. So I will take this opportunity to provide some basic "hardware" requirements for some of the SMDegrain variants :- Any BM3D CPU filters requires a CPU with AVX2 instructions, so that means basically most hi-end Ryzens, Threadrippers, etc, but very few Intel's, maybe Gen 12. I would suggest you check the specs of the CPU's you use !! Any BM3D CUDA filters requires an nVidia GPU (faster the better). Any DGDecNV filters requires an nVidia GPU for the nvenc/nvdec, functions. Any KNLMeansCL filters requires OpenCL, either nVidia or AMD GPU's. Most of the other filters are good to go. Also, if you want to suss out more "Dogway" modded SMDegrain stuff, go here :- https://forum.doom9.org/showthread.php?t=182881 The link you posted for SMDegrain info is now a little dated, due to Dogway's mod's. (that's what I use) And don't ignore some of the other filters in the Custom list... I would be shocked if Atak did do anything to implement some SMDegrain filters, but never, say never. And as for Auto Updates, just disable it, to avoid problems !!!! Cheers & thanks ![]() PS:- I have downloaded your Bladerunner sample, will take a close look.
__________________
Not poorly done, just doin' it my way !!! So much to do, and so little time :( |
|
![]() |
![]() |
![]() |
#19483 | Link | ||
Formally known as .......
Join Date: Sep 2021
Location: On a need to know basis.
Posts: 220
|
Quote:
I did some more testing on my "slow" dual E5-2697v2 yesterday, and tried all sorts of setting's to see if I could get some performance back. And I ended up changing a couple of x265 commands, and filter settings. I even tried NUMA & no NUMA, and with NUMA enabled, made a very noticeable speed increase, and if you only use 2 encoding port address's, you have one for /node 0, and one for /node 1...nice. Speaking on NUMA, Ryzens don't have it, BUT Threadripper's & EPYC's, apparently do.... And I still don't think SMDegrain needs to be added to the filter thread list, I actually noticed a slight increase in speed, with it not even IN the list, in the RipBot264.ini file. And I think I had a bit of a brain fart with the R720 the other day when I got that x265 error....as it had previously worked ok on an encode, but the next encode, I got the error...turns out I was using a different filter which required AVX2, which the Xeon's don't have..... Quote:
__________________
We all need to get a life, before it's over. Long term RipBot264 user. Ryzen 5950X, 5900X & 3950X. Dual Xeon E5-2697v2. 2 x Dell Poweredge R720's (dual E5-2680v2) and many others |
||
![]() |
![]() |
![]() |
#19484 | Link | |
Grumpy Old Man.
Join Date: Jul 2019
Location: Out There....
Posts: 643
|
Quote:
![]() You just need to convert the audio to E-AC3 @ 1536kbps. NICE, cheers.
__________________
Not poorly done, just doin' it my way !!! So much to do, and so little time :( |
|
![]() |
![]() |
![]() |
#19486 | Link |
Registered User
Join Date: Mar 2011
Posts: 349
|
My main RP machine is a virtual machine with 24 cores and 48 threads. The Linux KVM Hypervisor has dual E5-2680 v4 @ 2.40GHz (14 cores) and passes 24 cores and 48 threads to the VM. I have a first gen Ryzen 1800X that I use for an encoding server. With those I see 7-12 FPS with 4K just using SMDegrain CPU and no GPU assistance.
|
![]() |
![]() |
![]() |
#19487 | Link |
Registered User
Join Date: Mar 2011
Posts: 349
|
SMDegrain - Issue and Solution
So I encoded Invasion of the Body Snatchers 4K over last night and I started watching it this morning and I encountered a problem in one of the scenes in the movie.
It's a dark scene with two people talking and their faces are in the light. If you watch the women's cheek you will see that seems degrain for a quarter second, then not degrain, and back and forth. It was very odd. I extracted that 90 second scene out and looked it it in detail and the source has some odd grain around the women's cheek and that is probably where the problem occurs. I ran a bunch of tests using different options against the sample in order to find out how to solve the problem. Reducing the block size from the default of 16 to 8 solved the problem. As a bonus the file size was 11% smaller than before using the default block size. Encoding takes a hit since I'm essentially quadrupling the number of blocks to analyze. Encoding was about 30-40% slower. Link to the sample source and the two encoded files (link valid for two weeks): https://cloud.chrisdos.com/s/6JkTfF2etSo8Ln8 I've changed my scripts to use a block size of 8 now: SMDegrain(video,tr=8,thSAD=800,thSADC=400,thSCD2=80,contrasharp=true,prefilter=2,refinemotion=true,blksize=8) Pauly Dunne: You have a lot of scripts. Most would not run on my system since I don't have a GPU. So you may want to add a GPU/CPU in the name or provide a different folders with GPU and CPU optimized scripts. You may also want to condense the scripts into an advanced vs simplified. Perhaps just have 4-5 scripts from very light to very hard and then all the other scripts in a different folder for those that want to try all the different options and alternatives to SMDegrain. |
![]() |
![]() |
![]() |
#19488 | Link | |
Grumpy Old Man.
Join Date: Jul 2019
Location: Out There....
Posts: 643
|
Quote:
You're like a dog with a bone....VERY particular (which is good) Yes, I realise there are a LOT of scripts, and I doubt that many of them would get used, once the user has "favourite's", then they could quite easily copy the others to another folder, and greatly reduce the drop down list. As for using a different folder, for different type or category's, how would you "call them up" from within RipBot ??? The only way I've known is to have EVERYTHING in the Custom folder, so it is available with the "Custom Script" option "box". If you create a New Folder within the Custom Folder, it does not show up in the Custom Script list !!!! So if you know a way to do that, please let me know, and I will add it !!! Same goes for Advanced & Simplified, how ??? However, I do like your suggestion of using CPU or GPU in the scripts name ![]() Anyway, it's a good base for users to start with, and customise to meet their own requirements, and remove the rest. I have downloaded your 2 sample's, and I will have to run them on a PC, but I DID noticed they are a very high bitrate, coming from you (I know they are only tests), but the Blade Runner sample @ CQ18, was around 6700kb/s, and todays were in the low to mid 40mb/s. So keep up the good work, and if you have any suggestions / changes or additions, please let me know.....Upward & Onward ![]() Cheers & thanks.
__________________
Not poorly done, just doin' it my way !!! So much to do, and so little time :( |
|
![]() |
![]() |
![]() |
#19489 | Link |
Formally known as .......
Join Date: Sep 2021
Location: On a need to know basis.
Posts: 220
|
I stumbled across some new/old info about OpenCL support for Intel CPU's that maybe you could suss out (I know I'm going to try again).
Atak posted this on another Forum :- https://forum.doom9.org/showthread.p...19#post1969319 So I downloaded the Intel OpenCL Driver (needs an Intel account,tho). So if you wanted to try a KNLMeansCL script without a GPU, you could mod the script to use device_type="CPU".
__________________
We all need to get a life, before it's over. Long term RipBot264 user. Ryzen 5950X, 5900X & 3950X. Dual Xeon E5-2697v2. 2 x Dell Poweredge R720's (dual E5-2680v2) and many others |
![]() |
![]() |
![]() |
#19490 | Link | |
Registered User
Join Date: Mar 2011
Posts: 349
|
Quote:
Using CQ also allows me to judge how clean the output is. I had another 4K movie that looked pretty clean and just had some light grain and it came out to be 18GB in size using CQ18 with no degraining. Just adding a little MDegrain2 with a thSAD=100 brought the file size down to 8GB still with CQ18. There really is no downside to using constant quality. Only reason I see to use a hard bit rate is if I'm trying to fit it on a medium that has a finite size limitation. |
|
![]() |
![]() |
![]() |
#19491 | Link |
Registered User
Join Date: Mar 2011
Posts: 349
|
I think SMDegrain gives me the all the options I want at this point. The prefilter options and sharpening that actually works instead of adding artifacts like my tests gave me before. I'm very happy with SMDegrain at this point and I'll be creating scripts using it almost exclusively.
|
![]() |
![]() |
![]() |
#19492 | Link | |
Grumpy Old Man.
Join Date: Jul 2019
Location: Out There....
Posts: 643
|
Quote:
Hi, can you please comment on this... Thanks
__________________
Not poorly done, just doin' it my way !!! So much to do, and so little time :( |
|
![]() |
![]() |
![]() |
#19493 | Link |
Registered User
Join Date: Mar 2011
Posts: 349
|
Since you can't have subfolders and such, my suggestions would be something like the following:
In the custom scripts, have only about a dozen scripts with generic names, such as: SMDegrain_GPU_Light/Medium/Hard/Very_Hard SMDegrain_CPU_Light/Medium/Hard/Very_Hard Then in your install instructions, provide another folder with the advanced scripts and tell them them where to place them. I would also add the requirements in the script name, such as GPU, CPU, AVX2, etc. Then the install instructions can also have an advanced section and you can lay out what HQD3D (which I think can only do 8-bit color), CQTMGC, BM3D, etc, can do. Maybe even provide a link that has more details such as that link I provided for SMDegrain. In the beginning, all the scripts will overwhelm someone. There is also probably quite a bit of overlap. Such as is using BM3D going to give better results than just SMDegrain, and if so, what are the advantages/disadvantages. |
![]() |
![]() |
![]() |
#19494 | Link | |
Registered User
Join Date: Mar 2011
Posts: 349
|
SMDegrain - Issue and Another Solution
Quote:
|
|
![]() |
![]() |
![]() |
#19495 | Link | |
Grumpy Old Man.
Join Date: Jul 2019
Location: Out There....
Posts: 643
|
Quote:
I know there's a LOT that I personally wouldn't use. I'll have a good look at it, when I get the chance...maybe I'll get a "lightbulb" moment ![]()
__________________
Not poorly done, just doin' it my way !!! So much to do, and so little time :( |
|
![]() |
![]() |
![]() |
#19496 | Link | ||
Grumpy Old Man.
Join Date: Jul 2019
Location: Out There....
Posts: 643
|
Quote:
So, I can't help but notice you're are mainly using the Extra Hard (might have a name change soon), scripts, and only changing the "tr=" value, and maybe the pre-filter #. With the Light, Medium & Hard scripts the "tr=" value is changed, along with a lot of the "th" type values, have you tried any of them for lesser noisy sources ?? Also, I had another good look at the SMDegrain link's info, and it would be interesting to try this script :- Quote:
I think thSCD1 works in conjunction with blksize values. And you could leave thSCD2 out (like you mentioned) I wonder if there's some way to incorporate a 2 pass SMDegrain run, like here (and the post below):- https://forum.doom9.org/showthread.p...68#post1969268 Anyway, regardless to all that, I'm just so happy that you have finally come around to using & preferring SMDegrain, it was one aim of mine, when I started doing all these additions, so thankyou ![]() And please, if you come up with some different scripts or filters, don't forget to let me know, either here on the Forum, a PM, or email. PS:- It would be interesting to try these out on the Jetson's, and see what result you get, even though you're happy with how you did them, several months back.
__________________
Not poorly done, just doin' it my way !!! So much to do, and so little time :( |
||
![]() |
![]() |
![]() |
#19497 | Link | |
Registered User
Join Date: Mar 2011
Posts: 349
|
Quote:
But those super grainy sources always bothered me and I did not like using MDegrain3 because of the blurring I saw in skin textures when there was movement. SMDegrain seems to do really well and the results are natural looking. I have not tested any lighter grainy sources yet, but I feel it will at least do as well as MDegrain2. I probably won't be doing any blurring/sharpening on the lighter sources. I'll start with the tr=2 for lighter degrain and tr=4 probably for medium degrain. Strangely, the tr=2 would be the equivalent of MDegrain2 and tr=8 would be MDegrain8. But the file size difference between tr=2 and tr=8 was dramatic and I saw no blurring issues like when just using MDegrain3. Encoding performance is still good using tr=8. I tried tr=12 trying to solve the issue I mentioned but the slow down was not worth it. I have Superman The Animated Series still up on the server. I have a run with MDegrain2 and using Handbrake with NLMeans animation Tune. I'll run SMDegrain against it as well to see where it ends up and compare the three. Jetsons and Flintstones I'll probably just leave alone as the results from Handbrake are stunning, just the encoding was so slow it was just painful to do. |
|
![]() |
![]() |
![]() |
#19498 | Link | ||
Registered User
Join Date: Jan 2010
Posts: 425
|
Quote:
Quote:
__________________
E5 2697 v2 @ 3.0GHz on P9X79 Deluxe 24GB Xeon E5-2680 v2 @ 3.1GHz 16GB Sony Vaio VPC-F13Z1E/B |
||
![]() |
![]() |
![]() |
#19499 | Link |
Formally known as .......
Join Date: Sep 2021
Location: On a need to know basis.
Posts: 220
|
Hi slalom, you haven't posted in quite a while...
I kind of agree, but it's not always the best option. Depending on the workload.
__________________
We all need to get a life, before it's over. Long term RipBot264 user. Ryzen 5950X, 5900X & 3950X. Dual Xeon E5-2697v2. 2 x Dell Poweredge R720's (dual E5-2680v2) and many others |
![]() |
![]() |
![]() |
#19500 | Link |
Registered User
Join Date: Jan 2019
Posts: 5
|
Hi, Is there there a guide of some sort for the distributed encoding? i'm trying to figure why it isn't working for me.
Ive got encoding servers open on both PC's and both PC's Ip's entered into the encoding client on the main PC both say online, but when i add a movie to encode and hit start it just says "please wait" and then after a minute or two says "All jobs finished" pop up notification despite nothing actually happening. Ive got both PC's windows firewall disabled to eliminate that as the issue. I have read about some software causing issues which is why i reinstalled windows on one PC and the other received a fresh reinstall about three days ago and only has 6 programs installed since then (Plex, Sonarr, Radarr, SABnzbd, media info, 7zip) Is there a port that needs to be open? ive got no idea why this isnt working |
![]() |
![]() |
![]() |
Tags |
264, 265, appletv, avchd, bluray, gui, iphone, ipod, ps3, psp, ripbot264, x264 2-pass, x264 gui, x264_64, x265, xbox360 |
Thread Tools | Search this Thread |
Display Modes | |
|
|