Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 19th May 2020, 21:18   #1  |  Link
samf88
Registered User
 
Join Date: Oct 2006
Posts: 19
Nasty noise on a bad DVD transfer

Hi, hope someone out there can help me out.

I've ripped a DVD which looks shockingly bad and I was hoping there was something I could do to clean it a little. I fear it's too far gone and I'd end up decimating what little detail there is, but maybe there's a magic filter chain out there that's perfect for making this situation slightly more bearable? I think this will be one of those shows that was edited on video so a lot of the damage may be permanent.

I've been dabbling with MDegrain3 which does clear up small parts of the picture on default settings, but everything else (such as TBilateral, VagueDenoiser, RemoveGrain) has to be ramped up to insane settings to even touch the sides of this awful noise, probably because each speck is massive. Does it have a name? I thought originally it was chroma noise, but now I'm not so sure.

This is a particularly bad frame, just using MPEG2Source and QTGMC's Very Slow preset.


Here's a video excerpt cut from the VOB, incase it's easier to identify frame-by-frame: https://drive.google.com/open?id=1Sr...fF7Rj8qShxhu3h

If anyone can shed some light - or even recommend a filter/chain that may help in some way - I'd really appreciate it. This show has a very special place in my heart and I wish it was treated better for its (now out of print) DVD release.

I use AVISynth+ 64 bit, if that matters.
samf88 is offline   Reply With Quote
Old 20th May 2020, 11:38   #2  |  Link
tebasuna51
Moderator
 
tebasuna51's Avatar
 
Join Date: Feb 2005
Location: Spain
Posts: 5,883
For what do you use QTGMC?
It is not interlaced, and you can use the QTGMC denoisers directly to see what is your prefered.
__________________
BeHappy, AviSynth audio transcoder.
tebasuna51 is offline   Reply With Quote
Old 20th May 2020, 13:06   #3  |  Link
samf88
Registered User
 
Join Date: Oct 2006
Posts: 19
Hi - thanks for replying!

QTGMC is being used to deinterlace the TFF footage which contains visible combing. I did try with EZDenoise early on in my quest but unfortunately it didn't really affect this type of noise on sensible values, and of course high values just turn it to a blur. More advanced settings - such as switching to dfttest - also don't seem to touch it.

I'm guessing I would need to combine a filter with some kind of edge mask to try and retain some detail, but I highly doubt I'll be able to get good results from this.

Last edited by samf88; 20th May 2020 at 13:33.
samf88 is offline   Reply With Quote
Old 20th May 2020, 16:21   #4  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,286
Just throwing some scripts at it, I preferred the TemporalDegrains.

TemporalDegrain removes more noise than TemporalDegrain2, but it blurs more. You'd probably have to encode a bit to judge how it looks when the pictures are moving. MAA is from the AnimeIVTC script, or there's MAA2. It's one of real.finder's functions, as is DeHalo_alpha_mt
https://forum.doom9.org/showthread.php?t=174121

Some SMDegrain experts might come along with some clever denoising settings using it instead.

MAA()
TemporalDeGrain()
DeHalo_alpha_mt()

Source


TemporalDegrain


TemporalDegrain2

Last edited by hello_hello; 20th May 2020 at 16:25.
hello_hello is offline   Reply With Quote
Old 20th May 2020, 16:28   #5  |  Link
samf88
Registered User
 
Join Date: Oct 2006
Posts: 19
Thanks! That's wonderful, gives me something to build on. I really appreciate your help, thank you. :-)

If anyone else has any alternative suggestions I'd still be happy to hear them.

Last edited by samf88; 20th May 2020 at 17:43.
samf88 is offline   Reply With Quote
Old 21st May 2020, 17:36   #6  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,286
I had a silly idea. I can't say I've done it before myself (it'd be dog slow), and I don't know if there'd be a better way to go about it, but what if you only denoised the chroma, then denoised the chroma and luma? You could probably get away with less noise removal the second time around because it'd look more like grain. It's probably mainly an improvement for flat areas of color, such as the sky, and not so much everywhere else.

A = MAA()
MergeChroma(A, A.TemporalDegrain())
TemporalDegrain2()
DeHalo_alpha_mt()



You could do something like this, just to double denoise where it's needed.

A = MAA()
B = A.TemporalDeGrain2()
C = MergeChroma(A, A.TemporalDegrain()).TemporalDegrain2()

B.Trim(0,478) ++ C.Trim(479,683) ++ B.Trim(684,0)
DeHalo_alpha_mt()

Except the logic for running two instances of TemporalDeGrain2 in a script doesn't seem right. To make that work you need to change line 156 in the script to this:

ReplaceGlobals = true

Last edited by hello_hello; 21st May 2020 at 18:40.
hello_hello is offline   Reply With Quote
Old 21st May 2020, 19:33   #7  |  Link
Mounir
Registered User
 
Join Date: Nov 2006
Posts: 722
That should be more than enough to me:
Quote:
SetMemoryMax(1200)
SetMTMode(3,4)
MPEG2Source("tgft.d2v", cpu=0, info=3)
# Step 1: Remove chroma noise:
assumetff()
ConvertToYV16(interlaced=true)
orig=last
ev=orig.assumetff().separatefields().selecteven()
od=orig.assumetff().separatefields().selectodd()
ev
ue_chroma = UToY(ev).blur(0,1.5).binomialblur(5).ttempsmooth(maxr=2,lthresh=150, strength=1)
ve_chroma = VToY(ev).blur(0,1.5).binomialblur(5).ttempsmooth(maxr=2,lthresh=150, strength=1)
YToUV(ue_chroma, ve_chroma)
MergeLuma(ev)
ev_filtered=last
od
uo_chroma = UToY(od).blur(0,1.5).binomialblur(5).ttempsmooth(maxr=2,lthresh=150, strength=1)
vo_chroma = VToY(od).blur(0,1.5).binomialblur(5).ttempsmooth(maxr=2,lthresh=150, strength=1)
YToUV(uo_chroma, vo_chroma)
MergeLuma(od)
od_filtered=last
interleave(ev_filtered,od_filtered)
assumefieldbased().assumetff().weave()
# Step2: color correction
ConvertToRGB32(interlaced=true)
########
LoadVirtualDubPlugin("C:\Program Files (x86)\VirtualDub\plugins\CMYK_rj.vdf", "CMYK", 0)
CMYK(255,0,266,10,257,-24,255,0,0,42) # Modifying Yellow high tones (-24%) and magenta lows (4%)


https://www.hostpic.org/view.php?fil...0130260109.jpg

Last edited by Mounir; 21st May 2020 at 21:00.
Mounir is offline   Reply With Quote
Old 21st May 2020, 19:37   #8  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,857
QTGMC(preset="very slow", inputtype=2) should already help nicely as it will press down the sharp noise and remove some aliasing.
It's hard to remove all the noise without removing all the detail. If you like, you could hit it with something TemporalDegrain2 and apply some light artificial noise with GrainFactory to fake detail.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 21st May 2020, 22:16   #9  |  Link
samf88
Registered User
 
Join Date: Oct 2006
Posts: 19
Quote:
Originally Posted by hello_hello View Post
I had a silly idea. I can't say I've done it before myself (it'd be dog slow), and I don't know if there'd be a better way to go about it, but what if you only denoised the chroma, then denoised the chroma and luma? You could probably get away with less noise removal the second time around because it'd look more like grain. It's probably mainly an improvement for flat areas of color, such as the sky, and not so much everywhere else.
Thank you so much, I do really like those results. Thanks so much for your help

Quote:
Originally Posted by Mounir View Post
That should be more than enough to me:
Thanks so much for that. I've had to modify it slightly for 64 bit, but I do like the results. Still trying to find a 64 bit version of the VDub filter, but I tried it manually in VDub itself to see the results. Thanks for taking the time to work on that, I really appreciate it.

Quote:
Originally Posted by Boulder View Post
QTGMC(preset="very slow", inputtype=2) should already help nicely as it will press down the sharp noise and remove some aliasing.
It's hard to remove all the noise without removing all the detail. If you like, you could hit it with something TemporalDegrain2 and apply some light artificial noise with GrainFactory to fake detail.
Thanks - definitely thinking I will do pre-treatment with QTGMC before anything else. I will look in to GrainFactory - that's a good idea to reintroduce some "grain" to disguise other things.

Much appreciated everyone!

EDIT: Here's what I've been able to achieve so far, using the help provided here: http://www.framecompare.com/screensh...rison/99J2JNNU. I'm much happier with this than what I was getting a few days ago.

Last edited by samf88; 22nd May 2020 at 11:01. Reason: Adding current progress
samf88 is offline   Reply With Quote
Old 22nd May 2020, 10:50   #10  |  Link
Katie Boundary
Registered User
 
Katie Boundary's Avatar
 
Join Date: Jan 2015
Posts: 737
I swear by TNLmeans: http://avisynth.nl/index.php/TNLMeans

There's also a KNLmeans but I think it requires an NVIDIA GPU or something.
__________________
If I ask "How do I do X?" or "what happens if I do X?", and X is a very bad thing that no one would ever normally do, assume that I already know this, and that I have Katie reasons for asking anyway.
Katie Boundary is offline   Reply With Quote
Old 22nd May 2020, 11:03   #11  |  Link
samf88
Registered User
 
Join Date: Oct 2006
Posts: 19
Thanks! I actually use KNLMeansCL quite often but on its own it wasn't touching the main problem. My current filter chain (with results shown in the link I edited in my post above) does include it as part of a denoise mask.
samf88 is offline   Reply With Quote
Old 22nd May 2020, 11:57   #12  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,286
Quote:
Originally Posted by samf88 View Post
Thanks! I actually use KNLMeansCL quite often but on its own it wasn't touching the main problem. My current filter chain (with results shown in the link I edited in my post above) does include it as part of a denoise mask.
So don't keep us in suspense. What's the script you're using?
hello_hello is offline   Reply With Quote
Old 22nd May 2020, 12:42   #13  |  Link
samf88
Registered User
 
Join Date: Oct 2006
Posts: 19
Below is what I've got at the moment. Hopefully it all makes sense and I haven't made any silly mistakes. I'm using AvsPmod to preview/make changes which seems to crash every time I change a parameter, so it wasn't an easy task to adjust things. I haven't yet found an alternative filter to adjust the CMYK levels as suggested by Mounir. I'm also not confident in my grain settings and keep adjusting them. It's possible I've used the masks wrong too - I never was confident using those. I may have switched colour space more than necessary too. I've also taken out all the SetFilterMTMode/Prefetch stuff for the sake of posting here.

Code:
MPEG2Source("movie1.d2v", cpu=0)

# Mounir's Chroma Noise Removal
assumetff()
ConvertToYV16(interlaced=true)
orig=last
ev=orig.assumetff().separatefields().selecteven()
od=orig.assumetff().separatefields().selectodd()
ev
ue_chroma = UToY(ev).blur(0,1.5).binomialblur(5).ttempsmooth(maxr=2,lthresh=150, strength=1)
ve_chroma = VToY(ev).blur(0,1.5).binomialblur(5).ttempsmooth(maxr=2,lthresh=150, strength=1)
YToUV(ue_chroma, ve_chroma)
MergeLuma(ev)
ev_filtered=last
od
uo_chroma = UToY(od).blur(0,1.5).binomialblur(5).ttempsmooth(maxr=2,lthresh=150, strength=1)
vo_chroma = VToY(od).blur(0,1.5).binomialblur(5).ttempsmooth(maxr=2,lthresh=150, strength=1)
YToUV(uo_chroma, vo_chroma)
MergeLuma(od)
od_filtered=last
interleave(ev_filtered,od_filtered)
assumefieldbased().assumetff().weave()

## Mounir's Colour Correction
#ConvertToRGB32(interlaced=true)
#LoadVirtualDubPlugin("VirtualDub\x86\plugins32\CMYK_rj.VDF", "CMYK", 0)
#CMYK(255,0,266,10,257,-24,255,0,0,42) # Modifying Yellow high tones (-24%) and magenta lows (4%)

ConvertToYV12()

LoadPlugin("DeGrainMedian64\DeGrainMedian.dll")
Import("kirsch.avsi")
aux=DeGrainMedian(mode=0, limity=7, interlaced=true)
denoised = KNLMeansCL(d=4,a=4,s=6,h=2.0,channels="Y",rclip=aux)
edgemask1 = ConvertToRGB32(interlaced=true).Kirsch().ConvertToYV12(interlaced=true,matrix="PC.601")
edgemask2 = edgemask1.mt_binarize(120).Blur(1.0).Blur(1.0)
mt_merge(last,denoised,edgemask2)

# Deinterlace
QTGMC(preset="Very Slow", FPSDivisor=2)

# End filtering
DeHalo_alpha_mt()
Grainfactory3mod(g1str=2,g2str=2,g3str=2)

# Shifting image up 1px and cropping (I should probably do the shift and crop all in RGB32 colour space actually)
ConvertToRGB32(matrix="Rec601", interlaced=false)
Crop(0,1,0,0)
AddBorders(0,0,0,1)
ConvertToYV12()
Crop(10, 2, -18, -2)
FillMargins(0,1,0,1) # I like to keep to mod4 resolutions, so this just makes the extra 2px less obvious
samf88 is offline   Reply With Quote
Old 22nd May 2020, 12:58   #14  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,857
Your QTGMC call is incorrect. The sample was not interlaced like tebasuna51 mentioned. Set inputtype=2 instead of FPSDivisor if you want to see how it works on deinterlaced (as that's what your sample looks like) content.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 22nd May 2020, 13:07   #15  |  Link
samf88
Registered User
 
Join Date: Oct 2006
Posts: 19
Hi,

I've adjusted it now, thanks.
http://www.framecompare.com/image-co...rison/77DPWNNX
samf88 is offline   Reply With Quote
Old 22nd May 2020, 13:49   #16  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,857
Quote:
Originally Posted by samf88 View Post
You could also try inputtype=3, I couldn't tell which one looks better but type 2 seems worse in that sample screenshot.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 22nd May 2020, 14:18   #17  |  Link
samf88
Registered User
 
Join Date: Oct 2006
Posts: 19
Just gave it a go with inputtype=3 (http://www.framecompare.com/image-co...rison/99J21NNU). Judging by the pattern on the red blouse alone I'd say it looks best treating it as interlaced (inputtype=0 and discarding the dupe frames using FPSDivisor). The stripey top and pants definitely look better with inputtype=3 though.
samf88 is offline   Reply With Quote
Old 22nd May 2020, 14:52   #18  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,286
I use QTGMC in progressive mode quite a lot for denoising, but if you've already denoised and it's just for removing the combed edges...

I ask, as I can't run KNLMeansCL on this PC, so I haven't seen the final denoising, but InputType-2 throws away every second scanline, which is why it's fixing the jaggies. InputType=1 doesn't (I don't think), so it's not fixing the jaggies, but I wonder if MAA().QTGMC(InputType=1) might retain more detail. Just a thought.

A very minor nitpick...
I'm pretty sure the final cropping and resizing can be done this way to avoid the conversion to RGB and back. The resizer is just shifting the pixels by 1.

W = Width()
H = Height()
Spline36Resize(W, H, 0,1,W,H)
Crop(10, 2, -18, -2)
FillMargins(0,1,0,1)

Mind you, without resizing,
Crop(10,2,-18,-2) gets you to 1.32 with a SAR of 12:11
or
Crop(10,2,-18,-2) gets you to 1.29 with a SAR of 16:15

I'm sure that DVD would have a 12:11 (or ITU) SAR.
https://forum.doom9.org/showthread.p...27#post1058927

I'm a bit OCD about a minimum aspect ratio of 4:3, so I'd probably crop a few more pixels top and bottom to get there. I'd also crop a few more pixels left and right to get rid of the "halo line" on the left and make the right border cleaner.

The extra pixels you'll need to crop top and bottom for 4:3 wouldn't have been seen on a CRT due to over-scanning anyway, and a 4:3 picture fills a bit more of a 16:9 screen. That's just my personal preference though.

If you're interested, you can look at all that fun stuff with my script, whether you resize or not. It'll tell you what the final aspect ratio will be, and you can specify an aspect ratio for it to crop to, to see how much extra it'll crop etc.

CropResize(0,0, 16, 2, -20, -2, InSAR=12.0/11.0, CropDAR=4.0/3.0, NoResize=true, Info=true)

Last edited by hello_hello; 22nd May 2020 at 14:54.
hello_hello is offline   Reply With Quote
Old 22nd May 2020, 15:19   #19  |  Link
samf88
Registered User
 
Join Date: Oct 2006
Posts: 19
Quote:
Originally Posted by hello_hello View Post
I ask, as I can't run KNLMeansCL on this PC, so I haven't seen the final denoising, but InputType-2 throws away every second scanline, which is why it's fixing the jaggies. InputType=1 doesn't (I don't think), so it's not fixing the jaggies, but I wonder if MAA().QTGMC(InputType=1) might retain more detail. Just a thought.
There seems to be more jaggies doing it that way.



Quote:
Originally Posted by hello_hello View Post
A very minor nitpick...
I'm pretty sure the final cropping and resizing can be done this way to avoid the conversion to RGB and back. The resizer is just shifting the pixels by 1.

W = Width()
H = Height()
Spline36Resize(W, H, 0,1,W,H)
Crop(10, 2, -18, -2)
FillMargins(0,1,0,1)
Funny you should suggest that as I was originally using z_Spline36Resize to shift - I can't actually remember the reason why I switched to the colour space conversion method.

Quote:
Originally Posted by hello_hello View Post
I'm a bit OCD about a minimum aspect ratio of 4:3, so I'd probably crop a few more pixels top and bottom to get there. I'd also crop a few more pixels left and right to get rid of the "halo line" on the left and make the right border cleaner.
Annoyingly the video has wildly different cropping values throughout, I suspect due to being edited on video or something. I don't know if there's a filter to automatically detect and rearrange different crop values, but I'm not keen on the idea of overcropping - even though that halo line is really bugging me. I never resize either, unless I'm using nnedi_rpow2 or going from 1080p>720p, etc. I set SAR in the encoder as some devices don't seem to pick up setting a DAR in the container. I get what you mean though, almost like a mini version of when TV channels cropped 4:3 images to 14:9 to help during the AR transition era.

Quote:
Originally Posted by hello_hello View Post
If you're interested, you can look at all that fun stuff with my script, whether you resize or not. It'll tell you what the final aspect ratio will be, and you can specify an aspect ratio for it to crop to, to see how much extra it'll crop etc.

CropResize(0,0, 16, 2, -20, -2, InSAR=12.0/11.0, CropDAR=4.0/3.0, NoResize=true, Info=true)
Very fancy! I will keep a note of that for future use. Thanks so much for all your help.
samf88 is offline   Reply With Quote
Old 22nd May 2020, 15:27   #20  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,286
Maybe try MAA or MAA3 before denoising etc. Splitting the fields and denoising before putting them back together might effect it. I'm just theorizing...
hello_hello is offline   Reply With Quote
Reply

Tags
bad quality, dvd, noise

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 20:13.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.