Detail-safe anime derainbowing with MC&masked DFMDerainbow (3 or 5 radius)

mandarinka · 28th January 2015, 18:25

I have been using motion compensated and edge-masked DFMDeRaibow as my derainbower of choice for some time. At one point I tried to make it work with the fluxsmooth filter mod/alternative that has increased radius. Instead of just the previous and next frame being used, the previous two and next two frames are used now - so 5 frames are being used to temporally smooth the rainbows. This should in theory make the derainbowing more effective.
It is very simple modification of the original. But since I think that many people here might appreciate ready-made solutions with example how to use the MC and masking, here you have it to try.

Basically, this is nothing really new, I mostly post this so that random people looking for a derainbowing solution can find it here

The motion compensation is taken straight out of mvtools2 manpage's examples.

First the plain version:

To start, get DFMDerainbow from here: http://avisynth.nl/images/DFMDeRainbow-20140223.avsi
Actually I am lazy and use the old Masktoolsv1 version instead, but the functionality should be identical.

Code:

# your source here. Do inverse telecine first.
prerb = last
derbmask = last.tedgemask(threshY=7).mt_deflate()
prefiltered = last.FFT3Dfilter(sigma=1.5,bw=16,bh=16,bt=3,ow=8,oh=8,plane=0)
derbsuperfilt = MSuper(prefiltered) # all levels for MAnalyse
derbsuper = MSuper(levels=1) # one level is enough for MCompensate
derbbackward_vectors = MAnalyse(derbsuperfilt, truemotion = true, isb = true, chroma=false)
derbforward_vectors = MAnalyse(derbsuperfilt, truemotion = true, isb = false, chroma=false)
# use not-prefiltered (super) clip for motion compensation
derbforward_compensation = MCompensate(derbsuper, derbforward_vectors)
derbbackward_compensation = MCompensate(derbsuper, derbbackward_vectors)
# create interleaved 3 frames sequences
interleave(derbforward_compensation,last,derbbackward_compensation)
dfmderainbow(maskthresh=12)
selectevery(3,1) # return filtered central (not-compensated) frames only
mt_merge(prerb,last, derbmask,luma=true)
#you can uncomment this if you want to compare the look before and after in virtualdub or avsp
#return interleave(prerb, last)
#.
#.
#.
#profit...

And here is the 5-radius script:

First get the function and the flux5framesT function it uses: http://ulozto.net/xdyCnmhn/dfmderainbow5-zip alternatively: http://pastebin.com/FyXFaAcY http://pastebin.com/X1E7Urea

And then use this thing in your script.

Code:

# your source here. Do inverse telecine first.
prerb = last
derbmask = last.tedgemask(threshY=7).mt_deflate()
prefiltered = last.FFT3Dfilter(sigma=1.5,bw=16,bh=16,bt=3,ow=8,oh=8,plane=0)
derbsuperfilt = MSuper(prefiltered) # all levels for MAnalyse
derbsuper = MSuper(levels=1) # one level is enough for MCompensate
derbbackward_vec2 = MAnalyse(derbsuperfilt, isb = true, delta = 2, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4)
derbbackward_vec1 = MAnalyse(derbsuperfilt, isb = true, delta = 1, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4)
derbforward_vec1 = MAnalyse(derbsuperfilt, isb = false, delta = 1, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4)
derbforward_vec2 = MAnalyse(derbsuperfilt, isb = false, delta = 2, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4)
# use not-prefiltered (super) clip for motion compensation
derbforward_compensation1 = MCompensate(derbsuper, derbforward_vec1,thSCD2=160,thSCD1=600)
derbforward_compensation2 = MCompensate(derbsuper, derbforward_vec2,thSCD2=160,thSCD1=600)
derbbackward_compensation1 = MCompensate(derbsuper, derbbackward_vec1,thSCD2=160,thSCD1=600)
derbbackward_compensation2 = MCompensate(derbsuper, derbbackward_vec2,thSCD2=160,thSCD1=600)
# create interleaved 5 frames sequences
interleave(derbforward_compensation2, derbforward_compensation1, last, derbbackward_compensation1, derbbackward_compensation2)
DFMDeRainbow5(10)
#dfttest(Y = false) # you could uncomment this if you wanted to filter more strongly - but it leads to more discoloration
selectevery(5,2) # return filtered central (not-compensated) frames only
mt_merge(prerb,last, derbmask,luma=true)
#you can uncomment this if you want to compare the look before and after in virtualdub or avsp
#return interleave(prerb, last)
#.
#.
#.
#profilt...

FAQ...

Q: Why should I use this instead of Bifrost(), Derainbow() /joke/, or fft3dfilter()?
A: MC-ed and edge-masked DFMDerainbow is the most detail-safe and artifact-safe filter for this purpose that I know. It is also the highest quality. FFT3Dfilter discolors edges heavily, Bifrost() often causes gross combed artifacts in chroma (and fails to work in motion). DFMDerainbow has issues with color flashes due to fluxsmooth not being particularly intelligent, but when combined with motion compensation and edge-masking, it is the most behaved derainbower currently available (I might be wrong, but it seems nothing new/better has really happened in this space for the last 10 years). The quoted examples are quite safe to be used even for high quality encodes. In surveys, 9 out of 10 Iczers prefer motion-compensated and edgemasked DFMDerainbow() to competing products.

Q: I want to change the default behaviour/strength, what to do?
A: (1) I chose 12 as the default strength in DFDerainbow. Decrease that to get stronger filtering, increase to make it safer. Good values might be 7-15 I think?
(2) Another thing to try is to increase the edgemask sensitivity. In the tedgemask() call, adjust the threshY parameter. Lower values mean more stuff gets filtered and fainter edges get detected, higher values that the filtering gets applied to less places. Good values are 5 - 15, 7-10 being the recommended medium range I think.
(3) Third way to adjust the amount of filtering is to make the edgemask's lines more broad, to catch rainbows that bleed out of the lines: you can do this by following the tedgemask() by one or more mt_inflate() calls. This makes the lines thicker, so the filtered area is wider, but also this increases discoloration around edges. The opposite (limiting of the mask and filtering by making the mask lines thinner) can be done by mt_deflate().

Q: I want to use the simple DFMDerainbow() with the basic radius only (radius 3) instead of the radius 5 one. Is that okay?
A: 1-neighbor radius works pretty well too, that is what I usually use actually. It might be less effective for more persistent rainbowing tho.

Q: I want to have more derainbowing strength, even if it will discolor edges a bit more?
A: Sometimes improvement is possible when you insert additional filters into the motion-compensation stage, after the DFDerainbow() line. For example, dfttest(Y = false). This will cost you some quality as it discolors edges. You could also try calling DFMDerainbow twice there, but that is even more dangerous.

Q: What are the requirements?
A: Aside from the functions itself, you need Deen, MSharpen, Fluxsmooth, Masktools v2, Tedgemask and MVtools 2.

28th January 2015, 18:25	#1 \| Link
mandarinka Registered User Join Date: Jan 2007 Posts: 729	Detail-safe anime derainbowing with MC&masked DFMDerainbow (3 or 5 radius) I have been using motion compensated and edge-masked DFMDeRaibow as my derainbower of choice for some time. At one point I tried to make it work with the fluxsmooth filter mod/alternative that has increased radius. Instead of just the previous and next frame being used, the previous two and next two frames are used now - so 5 frames are being used to temporally smooth the rainbows. This should in theory make the derainbowing more effective. It is very simple modification of the original. But since I think that many people here might appreciate ready-made solutions with example how to use the MC and masking, here you have it to try. Basically, this is nothing really new, I mostly post this so that random people looking for a derainbowing solution can find it here The motion compensation is taken straight out of mvtools2 manpage's examples. First the plain version: To start, get DFMDerainbow from here: http://avisynth.nl/images/DFMDeRainbow-20140223.avsi Actually I am lazy and use the old Masktoolsv1 version instead, but the functionality should be identical. Code: # your source here. Do inverse telecine first. prerb = last derbmask = last.tedgemask(threshY=7).mt_deflate() prefiltered = last.FFT3Dfilter(sigma=1.5,bw=16,bh=16,bt=3,ow=8,oh=8,plane=0) derbsuperfilt = MSuper(prefiltered) # all levels for MAnalyse derbsuper = MSuper(levels=1) # one level is enough for MCompensate derbbackward_vectors = MAnalyse(derbsuperfilt, truemotion = true, isb = true, chroma=false) derbforward_vectors = MAnalyse(derbsuperfilt, truemotion = true, isb = false, chroma=false) # use not-prefiltered (super) clip for motion compensation derbforward_compensation = MCompensate(derbsuper, derbforward_vectors) derbbackward_compensation = MCompensate(derbsuper, derbbackward_vectors) # create interleaved 3 frames sequences interleave(derbforward_compensation,last,derbbackward_compensation) dfmderainbow(maskthresh=12) selectevery(3,1) # return filtered central (not-compensated) frames only mt_merge(prerb,last, derbmask,luma=true) #you can uncomment this if you want to compare the look before and after in virtualdub or avsp #return interleave(prerb, last) #. #. #. #profit... And here is the 5-radius script: First get the function and the flux5framesT function it uses: http://ulozto.net/xdyCnmhn/dfmderainbow5-zip alternatively: http://pastebin.com/FyXFaAcY http://pastebin.com/X1E7Urea And then use this thing in your script. Code: # your source here. Do inverse telecine first. prerb = last derbmask = last.tedgemask(threshY=7).mt_deflate() prefiltered = last.FFT3Dfilter(sigma=1.5,bw=16,bh=16,bt=3,ow=8,oh=8,plane=0) derbsuperfilt = MSuper(prefiltered) # all levels for MAnalyse derbsuper = MSuper(levels=1) # one level is enough for MCompensate derbbackward_vec2 = MAnalyse(derbsuperfilt, isb = true, delta = 2, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4) derbbackward_vec1 = MAnalyse(derbsuperfilt, isb = true, delta = 1, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4) derbforward_vec1 = MAnalyse(derbsuperfilt, isb = false, delta = 1, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4) derbforward_vec2 = MAnalyse(derbsuperfilt, isb = false, delta = 2, overlap=4, truemotion = true, chroma=true,search=5, searchparam=4) # use not-prefiltered (super) clip for motion compensation derbforward_compensation1 = MCompensate(derbsuper, derbforward_vec1,thSCD2=160,thSCD1=600) derbforward_compensation2 = MCompensate(derbsuper, derbforward_vec2,thSCD2=160,thSCD1=600) derbbackward_compensation1 = MCompensate(derbsuper, derbbackward_vec1,thSCD2=160,thSCD1=600) derbbackward_compensation2 = MCompensate(derbsuper, derbbackward_vec2,thSCD2=160,thSCD1=600) # create interleaved 5 frames sequences interleave(derbforward_compensation2, derbforward_compensation1, last, derbbackward_compensation1, derbbackward_compensation2) DFMDeRainbow5(10) #dfttest(Y = false) # you could uncomment this if you wanted to filter more strongly - but it leads to more discoloration selectevery(5,2) # return filtered central (not-compensated) frames only mt_merge(prerb,last, derbmask,luma=true) #you can uncomment this if you want to compare the look before and after in virtualdub or avsp #return interleave(prerb, last) #. #. #. #profilt... FAQ... Q: Why should I use this instead of Bifrost(), Derainbow() /joke/, or fft3dfilter()? A: MC-ed and edge-masked DFMDerainbow is the most detail-safe and artifact-safe filter for this purpose that I know. It is also the highest quality. FFT3Dfilter discolors edges heavily, Bifrost() often causes gross combed artifacts in chroma (and fails to work in motion). DFMDerainbow has issues with color flashes due to fluxsmooth not being particularly intelligent, but when combined with motion compensation and edge-masking, it is the most behaved derainbower currently available (I might be wrong, but it seems nothing new/better has really happened in this space for the last 10 years). The quoted examples are quite safe to be used even for high quality encodes. In surveys, 9 out of 10 Iczers prefer motion-compensated and edgemasked DFMDerainbow() to competing products. Q: I want to change the default behaviour/strength, what to do? A: (1) I chose 12 as the default strength in DFDerainbow. Decrease that to get stronger filtering, increase to make it safer. Good values might be 7-15 I think? (2) Another thing to try is to increase the edgemask sensitivity. In the tedgemask() call, adjust the threshY parameter. Lower values mean more stuff gets filtered and fainter edges get detected, higher values that the filtering gets applied to less places. Good values are 5 - 15, 7-10 being the recommended medium range I think. (3) Third way to adjust the amount of filtering is to make the edgemask's lines more broad, to catch rainbows that bleed out of the lines: you can do this by following the tedgemask() by one or more mt_inflate() calls. This makes the lines thicker, so the filtered area is wider, but also this increases discoloration around edges. The opposite (limiting of the mask and filtering by making the mask lines thinner) can be done by mt_deflate(). Q: I want to use the simple DFMDerainbow() with the basic radius only (radius 3) instead of the radius 5 one. Is that okay? A: 1-neighbor radius works pretty well too, that is what I usually use actually. It might be less effective for more persistent rainbowing tho. Q: I want to have more derainbowing strength, even if it will discolor edges a bit more? A: Sometimes improvement is possible when you insert additional filters into the motion-compensation stage, after the DFDerainbow() line. For example, dfttest(Y = false). This will cost you some quality as it discolors edges. You could also try calling DFMDerainbow twice there, but that is even more dangerous. Q: What are the requirements? A: Aside from the functions itself, you need Deen, MSharpen, Fluxsmooth, Masktools v2, Tedgemask and MVtools 2. Last edited by mandarinka; 28th January 2015 at 19:44.