Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 Doom9's Forum Simple MDegrain Mod - SMDegrain() Mod v1.8d
 Register FAQ Calendar Search Today's Posts Mark Forums Read

 24th December 2011, 14:20 #1  |  Link Dogway Registered User   Join Date: Nov 2009 Posts: 1,898 Simple MDegrain Mod - SMDegrain() Mod v1.8d EDIT (2021): New SMDegrain and more over this thread This is just a wrapper for mdegrain, nothing fancy or magical. Just convenient automatizations for common tasks to get things done fast and in good quality, without forgetting some flexibility too. I created this based off my needs, and evolved from there. I took Caroliano's smdegrain as a base to build it because I also wanted something simple to use and straight to the point. The main features of this function are: High bitdepth denoising Higher temporal radius (tr>3) Contrasharpening Interlaced support YUY2 support Motion Vectors Globals Input/Output RefineMotion Option (MRecalculate) +some more little options and any combination of the above supported. Everything is OFF by default so you just type smdegrain() and it works, but as soon as you start setting more parameters on you will find to need more plugins and scripts. To run it you will need at least mvtools2 and masktools2. Read on dependencies to know more. Introduction deserves a good read to understand not only the parameters but overall mvtools2 behaviour from what I have experiencied and read over the forums. This is my biggest function script I have done so I'm sure many things are wrong or can be improved for code efficiency, this is one reason I post it here. I'm not sure if it was right to add YUY2 and Globals support, as it adds lots of code without a meaningful gain on quality or possibilities. I might rethink its inclusion... I have to specially thank cretindesalpes for bearing my incesant questions related to this. Eagerly awaiting MT : ) enjoy! SMDegrain() Mod v1.8d (18-01-2012) Changelog Code:  v1.8d: - Added automatic dark protection for prefilter=3 (dfttest performs badly in darks) v1.71d - Fixed chroma variable when chroma=false, from 2 (copy chroma) to 1 (discard chroma) v1.71d - Explanation and cosmetics v1.71d - Added acknowledgments and updated explanation - Added brackets to all functions (it has been reported to cause problems on occasional circumstances) (http://forum.doom9.org/showthread.php?p=1515886#post1515886) - Reworked and fixed the parameters panel. Now you have a color guide to have an idea on what to tweak depending on selected mode in Show. For strings: "Speed", "Memory" and "Quality" - Optimized lsb_in for over 720p HD resolutions which was causing memory hogs - Fixed contrasharpening for YUY2 sources when lsb=true - Fixed some inconsistencies for YUY2 sources when prefilter=3 and extended protection a bit v1.7d: - Fixed and extended introduction explanation v.1.61d - Fixed some issues for Globals=1 (Read) in relation to lsb_in and luma expansion auto-prefilter v.1.61d - Added mild gauss blur prefilter option as prefilter=1. Consequent modes are displaced v.1.61d - Changed prefilter 1 and 2 to Minblur (Gauss/Median combination) v.1.63d - Updated MinBlur function v.1.62d and v.1.66 - Code tidied up v.1.62d - Updated luma expansion, now you can enhace darks (with 'Str' and 'Amp') in the same step as the TV->PC conversion stage, thus +optimized, +quality. v.1.63d by cretindesalpes (http://forum.doom9.org/showthread.ph...18#post1548318) - Removed pre_custom. Now use prefilter to load a prefiltered clip, as well as modes 0~3 - Added Show parameter. Now you can see the used and default parameters along the prefilter clip as well - Fixed one note related to prefilter clip input for interlaced content v1.6d: - Fixed and extended introduction explanation - Fixed Ditherpost for interlaced sources - Fixed a broken reference for reading MRecalculate globals when tr>3 - Fixed a Globals=1 MSuper automatization - Fixed a plane processing bypass when chroma=false - Added LSFmod as optional contrasharpening (slower, but biased contrasharpening and probably better depending on source) - Added default luma expansion auto-prefilter for motion search (16% more values) - Added YUY2 support - Added the 'mode' parameter of ditherpost() - Added support for interlaced temporal radius up to 6 v1.5d: - Changed "srchSuper" Global variable to "QTGMC_srchSuper", compatible with QTGMC. Anyhow you may probably want to create a new super clip. Test it out! v.1.41d - Introduction spelling and presentation v.1.42d - Added lsb and lsb_in for prefilter=2 (dfttest) v.1.43d & v.1.44d - Some better definitions for the settings help v.1.45d - pelclip is now subpixel = 3 ('sharp' parameter before) v.1.45d - Other minor tweaks v.1.45d - Fixed an important issue when using prefilter with subpixel=3 ('pelclip' parameter before) - Optimizations in ditherpost for chroma - Changed rfilter (hierarchical levels smoothing/scaling) from default 2 to 4. Better results in my judge - Some more little tweaks in code and introduction v1.4d: - Added pelclip option for top quality subpixel interpolation when pel > 1 - Changed QTGMCV parameter to Globals, and added a bunch of new features like vectors output from smdegrain() - Big introduction and code optimization tweaks v1.3d: - Fixed an important issue on the Interlaced parameter and improved overall handling (v.1.21d & v.1.3d) v1.2d: - Added Interlaced parameter, for processing interlaced sources - Some introduction and code optimization tweaks (v.1.12d & v.1.2d) v1.1d: - Implemented MDegrainN for tr > 3, from the MVTools2 mod of Dither - Minor introduction and code optimization tweaks v1.0d: - Added CClip parameter. Reference an earlier stage of your processing chain as your sharp version for the contrasharpening v0.9d: - Added QTGMCV parameter, for reusing vectors from QTGMC bob deinterlacer v0.8d: - Fixed a few things related to the lsb_in option - Minor introduction and code optimization tweaks v0.7d: - Implemented "fake" lsb_in option. Output will have slightly more quality (if any), and will compress better. (Experimental) - Fixed a not passed argument in the hpad, vpad parameters v0.6d: - Minor introduction and code optimization tweaks v0.5d: - Added dfttest option for preblur (now prefilter) option. Slower but better due to its internal deblock+denoise operation, plus sharper results - Added pre_custom option for the prefilter process, define here a denoised clip as your prefiltered version - Updated the introduction help v0.4d: - Added preblur option for slight vector blurring where motionmatch is bad (http://forum.doom9.org/showthread.php?t=161594) - Added RefineMotion parameter for better motion vectors. Based off Killer() (http://forum.doom9.org/showthread.ph...72#post1266572) - Added parameters "lsb" and "lsb_out", for the MVTools2 mod version of cretindesalpes' Dither (http://forum.doom9.org/showthread.ph...59#post1386559) - Other minor aesthetics and performance adjustments v0.3d: - Fixed plane (for MDegrain) and added chroma (for MAnalyse) parameters v0.2 : - Added introductory section - Added more parameters v0.1 : - Basic working version - Not released Last edited by Dogway; 7th October 2021 at 23:13.
 6th January 2012, 11:47 #2  |  Link Anacletus Registered User     Join Date: Mar 2005 Posts: 50 Hi Dogway, i'm getting an error while trying to use your function with these settings: Code: smdegrain(tr=6,thSAD=200,contrasharp=true,refinemotion=true,lsb=true, prefilter=3) The error is --> Any clues? If i use options 1 or 2 for the prefilter parameter it works, as well as i remove the parameter entirely.
 6th January 2012, 12:35 #3  |  Link Dogway Registered User   Join Date: Nov 2009 Posts: 1,898 It works for me in v.1.61, are you using this one? in v.1.6, prefilter=3 requires a pre_custom clip which must be supplied. It's the same as prefilter=4 in v.1.61 I might write some error proof code for this in later revisions. Anyway I recommend using the latest version as it fixes a few important things. If it's not this or you are using already v.1.61 tell me what is your source input for smdegrain. I work in avisynth 2.58 so I can't give advices if it's a v.2.6 related issue.
6th January 2012, 12:41   #4  |  Link
Anacletus
Registered User

Join Date: Mar 2005
Posts: 50
Quote:
 Originally Posted by Dogway It works for me in v.1.61, are you using this one?
Yes, meanwhile i found the issue, i had two different versions of the function in my plugin folder My bad. .

Leaving only the latest solved.

for the support and for the cool function

 14th January 2012, 10:10 #5  |  Link tormento Acid fr0g     Join Date: May 2002 Location: Italy Posts: 2,086 Ok, I have tried to apply SMDegrain multiple times to a noisy clip. Reference test from BD sample: Code: SetMTMode(5) SetMemoryMax(1000) LoadPlugin("D:\eseguibili\media\dgdecnv\DGDecodeNV.dll") DGSource("E:\in\2_28 Tora tora tora (extended japanese)\tora.dgi") CompTest(1) ChangeFPS(last,last,true) SetMTMode(2) SMDegrain (tr=6,PreFilter=1,contrasharp=false,lsb=true,mode=6) Output file: 61.971.031 ~ 5.87 fps First try: Code: SMDegrain (tr=6,PreFilter=1,contrasharp=false,lsb=true,mode=6) SMDegrain (tr=6,PreFilter=1,contrasharp=false,lsb=true,mode=6) Output file: 44.545.178 ~ 2.66 fps Second try: Code: SMDegrain (tr=6,PreFilter=1,contrasharp=false,lsb_out=true,Globals=2) # globals 2, process+export SMDegrain (tr=6, contrasharp=false,lsb_in =true,Globals=1,mode=6) Output file: 46.896.765 ~ 1.41 fps Well, I suppose the increased file size is because of less detail loss as same vectors are used but why the speed is decreased so much? __________________ @turment on Telegram Last edited by tormento; 14th January 2012 at 12:00. Reason: As 1.71d is out…
 14th January 2012, 17:37 #6  |  Link Dogway Registered User   Join Date: Nov 2009 Posts: 1,898 Code: SMDegrain (tr=6,PreFilter=1,lsb=true,mode=6) SMDegrain (tr=6,PreFilter=1,lsb=true,mode=6) 585 sec 0.560 fps 809 sec 0.404 fps 804 sec 0.406 fps Code: SMDegrain (tr=6,PreFilter=1,lsb_out=true) SMDegrain (tr=6, lsb_in =true,mode=6) 576 sec 0.567fps 928 sec 0.352fps 940 sec 0.348fps Code: SMDegrain (tr=6,PreFilter=1,lsb_out=true,Globals=2) SMDegrain (tr=6, lsb_in =true,Globals=1,mode=6) 259 sec 1.263fps (let's except this one since it was my first run after PC boot) 433 sec 0.755fps 581 sec 0.563fps I did 3 runs for each on a SD source. As you can see using globals is faster than without. In my case (probably not in yours) it is also faster than the double smdegrain line, it makes sense because in the second instance you don't do any internal(or external) prefilter and don't calculate any motion vectors, but I can't assure this for all the sources as there are many variables in play. But definitely faster than same code without globals. If you want to cut down times further try hpad=0, vpad=0, or chroma=false, although I had problems with this one recently. You can also try plane=0 (to process only luma) or lower the tr value to 3 or 4. Let me note that lsb_in is experimental code, use under test environment. Last edited by Dogway; 14th January 2012 at 17:56.
 14th January 2012, 19:47 #7  |  Link tormento Acid fr0g     Join Date: May 2002 Location: Italy Posts: 2,086 Would you please try with MT such as my script and possibly on a HD source? __________________ @turment on Telegram
 14th January 2012, 20:01 #8  |  Link Dogway Registered User   Join Date: Nov 2009 Posts: 1,898 My laptop is not too powerful core 2 duo 2.53Gh but I'm using multithreading, post a small snippet of the source to test
 14th January 2012, 23:26 #9  |  Link TheProfileth Leader of Dual-Duality     Join Date: Aug 2010 Location: America Posts: 134 Thankyou Dogway I am continually happy for your frequent improvement on this, specially since I actually am using this as a part of my newest filter that is to be released relatively soon-ish. __________________ I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood My New filter is in the works, and will be out soon
 15th January 2012, 12:14 #10  |  Link tormento Acid fr0g     Join Date: May 2002 Location: Italy Posts: 2,086 Here you are. This is a clip of the movie, not the CompTest I used. I'll post the results here as soon as I'll finish them. EDIT1: Ok, some results: PHP Code:  SetMTMode(5)SetMemoryMax(1000)LoadPlugin("D:\eseguibili\media\dgdecnv\DGDecodeNV.dll")DGSource("E:\raw\2_28 Tora tora tora (extended japanese)\tora.dgi")#CompTest(1)ChangeFPS(last,last,true)SetMTMode(2)SMDegrain (tr=6,PreFilter=1,lsb=true,mode=6)   39.500.299 ~ 7,44 fps PHP Code:  SetMTMode(5)SetMemoryMax(1000)LoadPlugin("D:\eseguibili\media\dgdecnv\DGDecodeNV.dll")DGSource("E:\raw\2_28 Tora tora tora (extended japanese)\tora.dgi")#CompTest(1)ChangeFPS(last,last,true)SetMTMode(2)SMDegrain (tr=6,PreFilter=1,lsb=true,mode=6)SMDegrain (tr=6,PreFilter=1,lsb=true,mode=6)   25.500.848 ~ 3,02 fps Your other two scripts doesn't almost even start… EDIT2: Even tried to use AviSynth 2.6MT. No success to encode the two "advanced" scripts. The working ones are a bit slower. __________________ @turment on Telegram Last edited by tormento; 15th January 2012 at 15:42.
 15th January 2012, 16:20 #11  |  Link Overdrive80 Anime addict     Join Date: Feb 2009 Location: Spain Posts: 675 Thanks for your work, is cool ^^ __________________ Intel i7-6700K + Noctua NH-D15 + Z170A XPower G. Titanium + Kingston HyperX Savage DDR4 2x8GB + Radeon RX580 8GB DDR5 + ADATA SX8200 Pro 1 TB + Antec EDG750 80 Plus Gold Mod + Corsair 780T Graphite
 15th January 2012, 18:52 #12  |  Link Dogway Registered User   Join Date: Nov 2009 Posts: 1,898 @Tormento: It's a memory hog produced by next line. (loading lsb_in) Code: Dither_merge16_8(inputY,output,DitherBuildMask(inputP.ditherpost(mode=-1,u=Chr,v=Chr),output.ditherpost(mode=-1,u=u,v=v))) As you see it worked for me for SD, it has problems for HD. The code is correct (although lsb_in is a hack) so it might work in more powerful systems. Anyways I have an alpha version of v.1.7.2 but I'm gonna wait until I find a workaround for this code to make it usable on HD sources in low end systems. There's also a possiblity the limiter is avisynth, I'm not sure. Despite this (and until I find a workaround) using tr=6 is probably too much and unnecesary for HD sources. cretindesalpes is working in making the motion analysis multithreading so we can expect a processing boost when he releases a new version of mvtools2 mod with this. Meanwhile the next lines are more realistic (although they still present the memory hog in my system): Code: Crop(0, 134, -0, -132) SMDegrain (tr=3,PreFilter=1,lsb_out=true,Globals=2,hpad=0,vpad=0,chroma=false,plane=0) SMDegrain (tr=3, lsb_in =true,Globals=1,hpad=0,vpad=0,chroma=false,plane=0,mode=6,show=false) (then process chroma spatially) Also DGSource might work better with setmtmode(2,x). Thanks for reporting these kind of issues, otherwise I wouldn't know what else to update, the function is pretty much finished to my eyes. Last edited by Dogway; 15th January 2012 at 19:01.
 15th January 2012, 19:44 #13  |  Link tormento Acid fr0g     Join Date: May 2002 Location: Italy Posts: 2,086 Dogway, I own a 2600k@5GHz with 16GB of ram. Not really a low end system __________________ @turment on Telegram
15th January 2012, 20:39   #14  |  Link
Dogway
Registered User

Join Date: Nov 2009
Posts: 1,898
Quote:
 Originally Posted by Dogway There's also a possiblity the limiter is avisynth, I'm not sure.
Then as I said it's avisynth. I have read it has an internal limit of 2Gb. Nothing else I can say or do besides supplying a hack workaround.

My 2 liners code is the same as the next block, so if that doesn't work, this won't either.

Code:
chroma=false
plane=0
chr=chroma?3:1
a=minblur().Dither_Luma_Rebuild(lsb   =true,uv=chr,S0=2.0)
vmulti = super_search.MAnalyse( multi=true,overlap=8,blksize=16,search=4,chroma=chroma,delta = 3)

inputY=last
ditherpost(mode=6)

ditherpost(mode=6)

 15th January 2012, 20:53 #15  |  Link tormento Acid fr0g     Join Date: May 2002 Location: Italy Posts: 2,086 The last resource would be get a x64 compile of the modified MVTools. __________________ @turment on Telegram
 15th January 2012, 21:11 #16  |  Link Dogway Registered User   Join Date: Nov 2009 Posts: 1,898 If avisynth has a limit of 2Gb I don't know how mvtools2 by being x64 is going to bypass that... I think its a limit of avisynth by design. But the problem in your case is not mvtools, the trigger of the memory hog is the Dither tools in the Dither_merge16_8() line. I will have a look to see what can I do in the next few days. For the time being just remove the lsb_out/lsb_in arguments for over 720p sources: Code: Crop(0, 134, -0, -132) SMDegrain (tr=3,PreFilter=1,lsb=true,Globals=2,hpad=0,vpad=0,mode=6) SMDegrain (tr=3, lsb=true,Globals=1,hpad=0,vpad=0,mode=6) This should work, even if you want to use tr=6. Last edited by Dogway; 15th January 2012 at 21:15.
16th January 2012, 21:10   #17  |  Link
tormento
Acid fr0g

Join Date: May 2002
Location: Italy
Posts: 2,086
Quote:
 Originally Posted by Dogway For the time being just remove the lsb_out/lsb_in arguments for over 720p sources
It works

26.020.775 ~ 3.87 fps
__________________
@turment on Telegram

 16th January 2012, 21:48 #18  |  Link mastrboy Registered User   Join Date: Sep 2008 Posts: 327 cool script, but not sure if the name fits anymore with "Simple"...
 18th January 2012, 10:01 #19  |  Link tormento Acid fr0g     Join Date: May 2002 Location: Italy Posts: 2,086 Ok, as x264 2145 is out, I have done the tests again with the clip I provided. SMDegrain is an alpha version of 1.8. The common part of the code is: PHP Code:  SetMTMode(3) SetMemoryMax(768) LoadPlugin("D:\eseguibili\media\dgdecnv\DGDecodeNV.dll") DGSource("E:\raw\2_28 Tora tora tora (extended japanese)\tora.dgi") ChangeFPS(last,last,true) SetMTMode(2)   Remaining lines and tests follows: PHP Code:  SMDegrain (tr=6,PreFilter=1,lsb=true,mode=6)   47.242.772 @ 7,08 fps ~ 100% CPU usage PHP Code:  SMDegrain (tr=6,PreFilter=1,lsb=true,mode=6) SMDegrain (tr=6,PreFilter=1,lsb=true,mode=6)   29.238.914 @ 2,91 fps ~ 50% CPU usage PHP Code:  SMDegrain (tr=6,PreFilter=1,lsb=true,Globals=2,hpad=0,vpad=0,mode=6) SMDegrain (tr=6,            lsb=true,Globals=1,hpad=0,vpad=0,mode=6)   29.893.549 @ 3,87 fps ~ 25% CPU usage PHP Code:  SMDegrain (tr=6,PreFilter=2,lsb=true,Globals=2,hpad=0,vpad=0,mode=6) SMDegrain (tr=6,            lsb=true,Globals=1,hpad=0,vpad=0,mode=6)   25.191.496 @ 3,70 fps ~ 25% CPU usage PHP Code:  SMDegrain (tr=6,PreFilter=1,lsb_out=true,Globals=2) SMDegrain (tr=6,            lsb_in =true,Globals=1,mode=6)   32.545.424 @ 3,62 fps ~ 25% CPU usage PHP Code:  SMDegrain (tr=6,PreFilter=2,lsb_out=true,Globals=2) SMDegrain (tr=6,            lsb_in =true,Globals=1,mode=6)   27.002.212 @ 3,31 fps ~ 25% CPU usage It's a pity for the decreasing CPU usage, perhaps it is a MT problem. P.S: Dogway, the 1.8 you posted is set to private. I am unable to download it. __________________ @turment on Telegram Last edited by tormento; 18th January 2012 at 10:19.

 Tags denoise, interlaced, mdegrain, smdegrain, yuy2