Moving dots I cant seem to get rid of

[DarkDream]

Ive seen and fixed plenty of dot crawl in sources but this really takes the cake at being a pain in the ass.

Perhaps its not dot crawl or something Ive not yet come across?

These lines during playback seem to have plenty of moving dots running along the lines. It looks pretty much like dot crawl to me but seems a little different.

I used the dot crawl filter checkmate() and an antialiaser and they still wont go away. They seem to be around a bit less afterwards though.

Its several times more noticeable in the DVD video source.

any suggestion?

Heres a sample vob clip unaltered video. I used DGIndex and saved project + demuxed video. Its 1 minute 28 seconds long.

They show up alot once it gets to around the 1 minute point and you see the guy wearing the red jacket. They show up like that in various places all through the entire video and DVD.

http://glumbouploads.com/0qoh16vn3j9p.html

Here are a few screenshots of the source after I have filtered it with checkmate() and an antialiaser.

[cretindesalpes]

Screenshots of a filtered video are almost useless to make a diagnostic. Please post an untouched sample of the DVD VOB.

[DarkDream]

Originally Posted by cretindesalpes

Screenshots of a filtered video are almost useless to make a diagnostic. Please post an untouched sample of the DVD VOB.

Its been so long since I needed to post an unaltered sample that I forget what I used to use.

was it dgindex?

how can I get you a vob file from the dvd without it being over 1GB?

[StainlessS]

Yes, just mark with the sliders in DGIndex.

EDIT: Then "Save Project".

[DarkDream]

Originally Posted by StainlessS

Yes, just mark with the sliders in DGIndex.

EDIT: Then "Save Project".

Alrighty, Ill work on getting the sample up asap if I can figure it out again. In the meantime heres a couple screenshots of the video unaltered coming from the original DVD. I hope these can help some. The moving dots are way worse before I filtered it.

[DarkDream]

Ok heres the sample clip requested. I used DGIndex and saved project + demuxed video. Its 1 minute 28 seconds long.

They show up alot once it gets to around the 1 minute point and you see the guy wearing the red jacket. They show up like that in various places all through the entire video and DVD.

http://glumbouploads.com/0qoh16vn3j9p.html

[DarkDream]

A shorter 30 second clip can be found here

http://glumbouploads.com/jq5qwkb0oplb.html

[librarian]

Try this script:

Mpeg2Source("C:\Script Avisynth\Sample\Sample\VTS_01_1.demuxed.d2v")

telecide()
decimate()
o=last
spline36resize(640,480)
removegrain(19,22)
bicubicresize(720,480)
awarpsharp2(128,chroma=3)
contrasharpening(last,o)
return last
 


function ContraSharpening (clip denoised, clip original)
{
s = denoised.minblur(1,1) # Damp down remaining spots of the denoised clip.
allD = mt_makediff(original,denoised) # The difference achieved by the denoising.
ssD = mt_makediff(s,s.removegrain(11,-1)) # The difference of a simple kernel blur.
ssDD = ssD.repair(allD,1) # Limit the difference to the max of what the denoising removed locally.
ssDD = SSDD.mt_lutxy(ssD,"x 128 - abs y 128 - abs < x y ?") # abs(diff) after limiting may not be bigger than before.

denoised.mt_adddiff(ssDD,U=2,V=2) # Apply the limited difference. (Sharpening is just inverse blurring.)

return( last )
}
# Nifty Gauss/Median combination
function MinBlur(clip clp, int r, int "uv")
{
uv = default(uv,3)
uv2 = (uv==2) ? 1 : uv
rg4 = (uv==3) ? 4 : -1
rg11 = (uv==3) ? 11 : -1
rg20 = (uv==3) ? 20 : -1
medf = (uv==3) ? 1 : -200


RG11D = (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2)
RG4D = (r==1) ? mt_makediff(clp,clp.removegrain(4,rg4),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.medianblur(2,2*medf,2*medf),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.medianblur(3,3*medf,3*medf),U=uv2,V=uv2)
DD = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
clp.mt_makediff(DD,U=uv,V=uv)
return(last)
}

[DarkDream]

Thanks for the script, Ill give it a try later today and post about how it worked.

Judging by the script you've given, it has something to do with grain and removing grain to get rid of those dots?

Is there a name or term for what that is in the video?

[librarian]

Originally Posted by DarkDream

Thanks for the script, Ill give it a try later today and post about how it worked.

Judging by the script you've given, it has something to do with grain and removing grain to get rid of those dots?

Is there a name or term for what that is in the video?

Removegrain doesn't remove grain only (with mode from 13 to 16 it is a deinterlacer!). I used RG mode 19 and 22 (blur and median blur), then contrasharpening to limit the blurring.
Sorry I dont know the name of this particular artifact.

[DarkDream]

Originally Posted by librarian

Try this script:

Mpeg2Source("C:\Script Avisynth\Sample\Sample\VTS_01_1.demuxed.d2v")

telecide()
decimate()
o=last
spline36resize(640,480)
removegrain(19,22)
bicubicresize(720,480)
awarpsharp2(128,chroma=3)
contrasharpening(last,o)
return last
 


function ContraSharpening (clip denoised, clip original)
{
s = denoised.minblur(1,1) # Damp down remaining spots of the denoised clip.
allD = mt_makediff(original,denoised) # The difference achieved by the denoising.
ssD = mt_makediff(s,s.removegrain(11,-1)) # The difference of a simple kernel blur.
ssDD = ssD.repair(allD,1) # Limit the difference to the max of what the denoising removed locally.
ssDD = SSDD.mt_lutxy(ssD,"x 128 - abs y 128 - abs < x y ?") # abs(diff) after limiting may not be bigger than before.

denoised.mt_adddiff(ssDD,U=2,V=2) # Apply the limited difference. (Sharpening is just inverse blurring.)

return( last )
}
# Nifty Gauss/Median combination
function MinBlur(clip clp, int r, int "uv")
{
uv = default(uv,3)
uv2 = (uv==2) ? 1 : uv
rg4 = (uv==3) ? 4 : -1
rg11 = (uv==3) ? 11 : -1
rg20 = (uv==3) ? 20 : -1
medf = (uv==3) ? 1 : -200


RG11D = (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2)
RG4D = (r==1) ? mt_makediff(clp,clp.removegrain(4,rg4),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.medianblur(2,2*medf,2*medf),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.medianblur(3,3*medf,3*medf),U=uv2,V=uv2)
DD = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
clp.mt_makediff(DD,U=uv,V=uv)
return(last)
}

It blurred the video terribly and made alot of the black lines very thin making it look terrible in comparison.

The ending result makes the video look like shit to where its far from acceptable in quality but it did get rid of the dots.

[Reel.Deel]

Hello DarkDream, this maybe overkill but you may want to try this. It's slow but is does seem to get rid of *almost* all artifacts. The orginal post of this script can be found here.

# Required plugins:
# -----------------
# dgdecode.dll
# TIVTC.dll
# mt_masktools.dll
# removegrain.dll
# repair.dll
# frfun7.dll
# checkmate.dll

mpeg2source("VTS_01_1.demuxed.d2v")

checkmate()

borders = bicubicresize(width()/2,height()/2).bicubicresize(width()+8,height())
stackhorizontal(borders.crop(0,0,4,0),last,borders.crop(width()+4,0,0,0))

o=last ox=o.width() oy=o.height()

drr     = o.separatefields().vinverse2().weave()
both    = stackhorizontal(drr,o)
IVTC    = both.tfm(pp=0,clip2=stackhorizontal(drr,drr)).tdecimate(mode=1)
raw     = IVTC.crop(ox,0,ox,oy)
clean1  = IVTC.crop(0,0,ox,oy)

D1      = mt_makediff(raw,clean1)
D8      = D1.bicubicresize(ox/2-72,oy).blur(1,0).bicubicresize(ox,oy,1,0)
D9      = mt_lutxy(D1,D8,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
clean1a = clean1.mt_adddiff(D9,U=2,V=2)
clean1b = clean1a.frfun7(1.01,8,1)

allD    = mt_makediff(raw,clean1b).greyscale()
shrpD   = mt_makediff(clean1b,clean1b.removegrain(20))
DD      = shrpD.repair(allD,13,0).mt_lutxy(shrpD,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
clean1c = clean1b.mt_lutxy(clean1a,"x 3 + y < x 2 + x y < x 1 + x 3 - y > x 2 - x y > x 1 - x ? ? ? ?",U=2,V=2)
                \.mt_adddiff(DD.sbr(),U=2,V=2)

raw     = raw.crop(4,0,-4,0,true)
clean1c = clean1c.crop(4,0,-4,0,true)

#interleave(raw,clean1c)   # interleave "only-IVTC" with result of full filterchain, for frame comparison
# clean1c                 # just result
return(clean1c)

#--- end of script ---#

#######################################################

#--- Helper functions below ---#

function Vinverse2(clip clp, float "sstr", int "amnt", int "uv")
{
uv   = default(uv,3)
sstr = default(sstr,2.7)
amnt = default(amnt,255)
uv2  = (uv==2) ? 1 : uv
STR  = string(sstr)
AMN  = string(amnt)
vblur  = clp.sbrV()
vblurD = mt_MakeDiff(clp,vblur,U=uv2,V=uv2)
Vshrp  = mt_LutXY(vblur,vblur.mt_convolution("1","1 2 1"),expr="x x y - "+STR+" * +",U=uv2,V=uv2)
VshrpD = mt_MakeDiff(Vshrp,vblur,U=uv2,V=uv2)
VlimD  = mt_LutXY(VshrpD,VblurD,expr="x 128 - y 128 - * 0 < x 128 - abs y 128 - abs < x y ? 128 - 0.25 * 128 + x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
mt_AddDiff(Vblur,VlimD,U=uv,V=uv)
(amnt>254) ? last : (amnt==0) ? clp : mt_LutXY(clp,last,expr="x "+AMN+" + y < x "+AMN+" + x "+AMN+" - y > x "+AMN+" - y ? ?",U=uv,V=uv) 
return(last)
}

function sbr(clip o) {
rg11=o.removegrain(11)
rg11D=mt_makediff(o,rg11)
rg11DD=mt_makediff(rg11D,rg11D.removegrain(11)).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
o.mt_makediff(rg11DD,U=2,V=2)
}

function sbrV(clip o) {
rg11=o.mt_convolution("1","1 2 1")
rg11D=mt_makediff(o,rg11)
rg11DD=mt_makediff(rg11D,rg11D.mt_convolution("1","1 2 1")).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
o.mt_makediff(rg11DD,U=2,V=2)
}

[lansing]

well the basic concept of removing the dots is, mask out all the lines(edges) in the image, and then apply blur to the lines to a point where the dots blend into a line, and then merge the mask back to the image.
Unfortunately, the edge masking filter is not smart enough to filter only the line with dots, so you'll end up with a blurrier image overall, with the "good" areas also blurred (smoothed) out.

[sirt]

Do you think you can make it "smarter" in this case ?

[DarkDream]

None of the above scripts worked well enough. They got rid of the dots but it made the video look like shit so I played around with my own combination of things and didnt even bother with the above submissions.

Actually, I finally got it quite well and I didnt even have to ruin much if anything that I can see. I dont notice anything going bad about it really. I didnt need any masking that I know of and I didnt really blur the image.

Heres some comparison screenshots.

(This will be much easier for you to see changes if you save all the pictures and then flip back and forth between them on your computer one right after another. Then you can see the changes easily because the picture wont be moving and will be switching back and forth between them)


TOP = My original encode that I posted shots of with dots
BOTTOM = The one I just did last night that I was going to see how it turned out


Ok in this first example, 99% of the dots on the shoulders are gone and I didnt really notice anything ruined about it. The new one even looks a bit sharper I think. If you look around the eyes, it looks like the new one might have darkened/thickened black lines in some areas of the video a tiny bit but no one will really notice during playback unless they had the original footage to compare to it. I could barely tell in the screenshots.






Thats much better than the original footage as you can see





In this next example again 99% of the dots are gone and I dont really see anything gone bad yet. Again I think I see black lines barely getting thicker/darker in some spots but without footage to compare and being picky I dont really think it means too much or is a bad thing. The only real change I notice between the two besides the dots gone is the bottom left corner of that opened door when I flip back and forth between the two quickly.

[Guest]

So what was your script that produced these results?

[DarkDream]

Originally Posted by neuron2

So what was your script that produced these results?

I added in

checkmate(tthr2=0) ------- [[[did (tthr2=0) so artifacts wouldnt show up during rapid movement]]]

QTGMC(preset="Slow", InputType=1) ------- [[[This is what I added in last night that took away the rest of the dots with ease]]]

and I added an antialiaser of my choice, a denoiser of my choice, and a sharpener of my choice at the end to get rid of the rest of the dots and to sharpen the image.

The antialiaser and QTGMC are what got rid of them completely. In my first encode I only had the antialiaser and checkmate without QTGMC. They reduced them alot but didnt fully get rid of them alone.

[Guest]

Can you please post the whole script for us?

[DarkDream]

Originally Posted by neuron2

Can you please post the whole script for us?

LoadPlugin("C:\Program Files (x86)\megui\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("C:\Documents and Settings\Administrator\Desktop\Zoids New Century Zero\Disc 1\Episode 1\VTS_01_1.d2v", info=3)
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)
checkmate(tthr2=0)
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\TIVTC.dll")
tfm(order=-1).tdecimate(mode=1,hybrid=1)
QTGMC(preset="Slow", InputType=1)
crop( 8, 0, -8, 0)
LanczosResize(640,480) # Lanczos (Sharp)
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\UnDot.dll")
Undot() # Minimal Noise
[[[denoiser of my choice]]]
[[[antialiaser of my choice]]]
[[[sharpener of my choice]]]


thats the entire thing

[DarkDream]

Originally Posted by neuron2

Can you please post the whole script for us?

DGDecode_mpeg2source("C:\Documents and Settings\Administrator\Desktop\Zoids New Century Zero\Disc 1\Episode 1\VTS_01_1.d2v", info=3)
ColorMatrix(hints=true, interlaced=true, threads=0)
checkmate(tthr2=0)
tfm(order=-1).tdecimate(mode=1,hybrid=1)
QTGMC(preset="Slow", InputType=1)
crop( 8, 0, -8, 0)
LanczosResize(640,480) # Lanczos (Sharp)
Undot() # Minimal Noise
[[[denoiser of my choice]]]
[[[antialiaser of my choice]]]
[[[sharpener of my choice]]]


thats the entire thing