Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 28th January 2015, 18:25   #1  |  Link
mandarinka
Registered User
 
mandarinka's Avatar
 
Join Date: Jan 2007
Posts: 729
Detail-safe anime derainbowing with MC&masked DFMDerainbow (3 or 5 radius)

I have been using motion compensated and edge-masked DFMDeRaibow as my derainbower of choice for some time. At one point I tried to make it work with the fluxsmooth filter mod/alternative that has increased radius. Instead of just the previous and next frame being used, the previous two and next two frames are used now - so 5 frames are being used to temporally smooth the rainbows. This should in theory make the derainbowing more effective.
It is very simple modification of the original. But since I think that many people here might appreciate ready-made solutions with example how to use the MC and masking, here you have it to try.

Basically, this is nothing really new, I mostly post this so that random people looking for a derainbowing solution can find it here The motion compensation is taken straight out of mvtools2 manpage's examples.


First the plain version:

To start, get DFMDerainbow from here: http://avisynth.nl/images/DFMDeRainbow-20140223.avsi
Actually I am lazy and use the old Masktoolsv1 version instead, but the functionality should be identical.

Code:
# your source here. Do inverse telecine first.
prerb = last
derbmask = last.tedgemask(threshY=7).mt_deflate()
prefiltered = last.FFT3Dfilter(sigma=1.5,bw=16,bh=16,bt=3,ow=8,oh=8,plane=0)
derbsuperfilt = MSuper(prefiltered) # all levels for MAnalyse
derbsuper = MSuper(levels=1) # one level is enough for MCompensate
derbbackward_vectors = MAnalyse(derbsuperfilt, truemotion = true, isb = true, chroma=false)
derbforward_vectors = MAnalyse(derbsuperfilt, truemotion = true, isb = false, chroma=false)
# use not-prefiltered (super) clip for motion compensation
derbforward_compensation = MCompensate(derbsuper, derbforward_vectors)
derbbackward_compensation = MCompensate(derbsuper, derbbackward_vectors)
# create interleaved 3 frames sequences
interleave(derbforward_compensation,last,derbbackward_compensation)
dfmderainbow(maskthresh=12)
selectevery(3,1) # return filtered central (not-compensated) frames only
mt_merge(prerb,last, derbmask,luma=true)
#you can uncomment this if you want to compare the look before and after in virtualdub or avsp
#return interleave(prerb, last)
#.
#.
#.
#profit...


And here is the 5-radius script:

First get the function and the flux5framesT function it uses: http://ulozto.net/xdyCnmhn/dfmderainbow5-zip alternatively: http://pastebin.com/FyXFaAcY http://pastebin.com/X1E7Urea

And then use this thing in your script.
Code:
# your source here. Do inverse telecine first.
prerb = last
derbmask = last.tedgemask(threshY=7).mt_deflate()
prefiltered = last.FFT3Dfilter(sigma=1.5,bw=16,bh=16,bt=3,ow=8,oh=8,plane=0)
derbsuperfilt = MSuper(prefiltered) # all levels for MAnalyse
derbsuper = MSuper(levels=1) # one level is enough for MCompensate
derbbackward_vec2 = MAnalyse(derbsuperfilt, isb = true, delta = 2, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4)
derbbackward_vec1 = MAnalyse(derbsuperfilt, isb = true, delta = 1, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4)
derbforward_vec1 = MAnalyse(derbsuperfilt, isb = false, delta = 1, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4)
derbforward_vec2 = MAnalyse(derbsuperfilt, isb = false, delta = 2, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4)
# use not-prefiltered (super) clip for motion compensation
derbforward_compensation1 = MCompensate(derbsuper, derbforward_vec1,thSCD2=160,thSCD1=600)
derbforward_compensation2 = MCompensate(derbsuper, derbforward_vec2,thSCD2=160,thSCD1=600)
derbbackward_compensation1 = MCompensate(derbsuper, derbbackward_vec1,thSCD2=160,thSCD1=600)
derbbackward_compensation2 = MCompensate(derbsuper, derbbackward_vec2,thSCD2=160,thSCD1=600)
# create interleaved 5 frames sequences
interleave(derbforward_compensation2, derbforward_compensation1, last, derbbackward_compensation1, derbbackward_compensation2)
DFMDeRainbow5(10)
#dfttest(Y = false) # you could uncomment this if you wanted to filter more strongly - but it leads to more discoloration
selectevery(5,2) # return filtered central (not-compensated) frames only
mt_merge(prerb,last, derbmask,luma=true)
#you can uncomment this if you want to compare the look before and after in virtualdub or avsp
#return interleave(prerb, last)
#.
#.
#.
#profilt...

FAQ...

Q: Why should I use this instead of Bifrost(), Derainbow() /joke/, or fft3dfilter()?
A: MC-ed and edge-masked DFMDerainbow is the most detail-safe and artifact-safe filter for this purpose that I know. It is also the highest quality. FFT3Dfilter discolors edges heavily, Bifrost() often causes gross combed artifacts in chroma (and fails to work in motion). DFMDerainbow has issues with color flashes due to fluxsmooth not being particularly intelligent, but when combined with motion compensation and edge-masking, it is the most behaved derainbower currently available (I might be wrong, but it seems nothing new/better has really happened in this space for the last 10 years). The quoted examples are quite safe to be used even for high quality encodes. In surveys, 9 out of 10 Iczers prefer motion-compensated and edgemasked DFMDerainbow() to competing products.

Q: I want to change the default behaviour/strength, what to do?
A: (1) I chose 12 as the default strength in DFDerainbow. Decrease that to get stronger filtering, increase to make it safer. Good values might be 7-15 I think?
(2) Another thing to try is to increase the edgemask sensitivity. In the tedgemask() call, adjust the threshY parameter. Lower values mean more stuff gets filtered and fainter edges get detected, higher values that the filtering gets applied to less places. Good values are 5 - 15, 7-10 being the recommended medium range I think.
(3) Third way to adjust the amount of filtering is to make the edgemask's lines more broad, to catch rainbows that bleed out of the lines: you can do this by following the tedgemask() by one or more mt_inflate() calls. This makes the lines thicker, so the filtered area is wider, but also this increases discoloration around edges. The opposite (limiting of the mask and filtering by making the mask lines thinner) can be done by mt_deflate().

Q: I want to use the simple DFMDerainbow() with the basic radius only (radius 3) instead of the radius 5 one. Is that okay?
A: 1-neighbor radius works pretty well too, that is what I usually use actually. It might be less effective for more persistent rainbowing tho.

Q: I want to have more derainbowing strength, even if it will discolor edges a bit more?
A: Sometimes improvement is possible when you insert additional filters into the motion-compensation stage, after the DFDerainbow() line. For example, dfttest(Y = false). This will cost you some quality as it discolors edges. You could also try calling DFMDerainbow twice there, but that is even more dangerous.

Q: What are the requirements?
A: Aside from the functions itself, you need Deen, MSharpen, Fluxsmooth, Masktools v2, Tedgemask and MVtools 2.

Last edited by mandarinka; 28th January 2015 at 19:44.
mandarinka is offline   Reply With Quote
Old 4th October 2015, 01:07   #2  |  Link
mandarinka
Registered User
 
mandarinka's Avatar
 
Join Date: Jan 2007
Posts: 729
Apparently I missed one dependency of Flux5frames.avsi.

For use of the 5-radius script, you need to have tmedian2() too. Save this as median2.avsi and put it to your plugin folder: http://pastebin.com/D7pUCKz8
mandarinka is offline   Reply With Quote
Old 8th October 2015, 05:19   #3  |  Link
bxyhxyh
Registered User
 
Join Date: Dec 2011
Posts: 354
I have a question. Why prefilter is denoising luma? (plane=0)
bxyhxyh is offline   Reply With Quote
Old 11th October 2015, 09:46   #4  |  Link
mandarinka
Registered User
 
mandarinka's Avatar
 
Join Date: Jan 2007
Posts: 729
The prefilter is only there to make the motion search easier/more effective (it's more or less a cargo cult bit, it could probably be improved). Its results are not used for output.
mandarinka is offline   Reply With Quote
Old 12th October 2015, 09:17   #5  |  Link
bxyhxyh
Registered User
 
Join Date: Dec 2011
Posts: 354
But don't MAnalyse analyze chroma and luma separately?
I don't know how it works internally, i'm not a programmer or something.

I just thought it won't affect to chroma analysis at all, since rainbow is chroma problem.
bxyhxyh is offline   Reply With Quote
Old 12th October 2015, 13:22   #6  |  Link
mandarinka
Registered User
 
mandarinka's Avatar
 
Join Date: Jan 2007
Posts: 729
Chroma isn't that important here, actually. The motion vectors will be found (if the search is successful) based on luma, mainly. Actually, in this case your chroma is quite problematic because of the rainbows, so searching based on it might not even help at all. It could make sense to exclude chroma search explicitly, so that the rainbows don't fool it.

In practice, I don't think disabling chroma search helps that much (as you can see, the 3-radius version I posted has it off, the 5-radius on - to be honest, I don't recall why I set it up that way, any more). You can try what happens on particular sources if you turn it on and off with chroma=true/false in the MAnalyse() calls. And/or you can try enabling chroma prefiltering by the FFT3D. The effects are probably mixed as said, because there are two conflicting factors: the extra detail in chroma can give you more precise/real motion vectors and better motion compensation, but the rainbowing can also confuse MVTools and cause the opposite, worse results.
mandarinka is offline   Reply With Quote
Old 2nd July 2017, 12:42   #7  |  Link
SaurusX
Registered User
 
Join Date: Feb 2017
Posts: 107
I'm bumping this because I just found it and it's extremely useful for those "burned in" rainbows not introduced by the telecine process. I found that using bifrost1.1 right after this process really helps to kill those final lingering rainbows on static scenes. With 1.1 you don't have the problem of chroma being displaced to the prior frame during fast motion like with bifrost2.0 since it either works on the entire frame or doesn't.

I never could get the 5-frame version of this to work properly, though.

I found this process joined with other types of temporal frame analysis will cause the video to go absolutely haywire. The mask pops through, lots of frame flashing, and lagged chroma in the final encode, so watch out for that.
SaurusX is offline   Reply With Quote
Reply

Tags
derainbow, derainbowing, rainbow

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 02:57.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.