GenMotion and DePan plugins for global motion compensation

[morsa]

Well, somebody tell me I'm terribly wrong, please.
I guess, seeing that a good motion estimation is on its way, that in a relative near future we will have the possibility of making a good frame synthesizer.Is it OK?

[SoonUDie]

Originally posted by morsa
Well, somebody tell me I'm terribly wrong, please.
I guess, seeing that a good motion estimation is on its way, that in a relative near future we will have the possibility of making a good frame synthesizer.Is it OK?

If what you mean is that we'll be able to accurately interpolate any framerate, then yes... and no. I believe some very good solutions already exist... however, they're many hundreds of dollars. The key is to relatively accurately find motion vectors for every pixel in the image.

However, while we could interpolate something like a 24FPS movie to 60FPS, how much do we really gain? We're not really re-creating the motion here... it's like taking an antiderivative - you have a +c floating around, and we don't know what that is.

For example, take the motion of a really fast moving ball. If we capture it at 30fps, it'll be a blur. At 60fps, it will be more like a ball, but still blurry. At 120fps, it'll look even more like a ball. Now let's say we interpolate that 30fps video up to 120fps. It won't look like a ball - you'll still have the blury 30fps picture - but it'll move 4x smoother: it'll traverse only 1/4 the pixels per frame as it did before.

I'd still like to see it happen, though, and preferably in real time - we live in a world where 60p is finally becoming the standard, and I'm starting to get tired of 24fps...

[morsa]

Nice answer
One idea I have would be to use it to recover lost frames, may be 2 or three in a row, or to convert from 24 fps to 30, 25 to 30, or better, 50 fields to 60...Who knows, there are so many options.....

[mf]

Most interesting is to synthesize frames that are in an odd position between frames, e.g. convert 30fps to 24fps, without blending.

[Fizick]

1. MF answer is nice, but not quite correct.
If we capture fast moving ball at 30fps, it'll NOT must be a blur.
Camera's (camcorder) frame shooting time (exposition) is dependent on record mode ("normal", "sport", "golf"). This time of course is not large than 1/fps (1/60 sec), but may be quite small for bright scenes (up to 1/2000 sec). The fast moving ball may be quite sharp, and framerate interpolation (with motion compensation) will be sharp too.

2. GenMotion may be used, for example, to change framerate with factor=1.5, from 16.6 fps (old 8 mm film) to 25 fps.

Script:

Avisource("kino.avi")
loadplugin("avisynth_c.dll")
converttoYV12()
loadCplugin("genmotion.dll")
i=ConvertToYV12()
f1_3=GenMotion(i,filename="c:\deshaker.log",delta=1./3)
b1_3=GenMotion(i,filename="c:\deshaker.log",delta=-1./3)
interleave(f1_3,i,b1_3)
selectevery(6,0,1,2)
Crop(48,32,720-48-32,576-32-32).AddBorders(48,32,32,32)
converttoYUY2()

[SoonUDie]

Originally posted by Fizick
If we capture fast moving ball at 30fps, it'll NOT must be a blur.

Yea, I was just using that as an example. Besides, any object of any speed will have varying motion blur depending on how close it is to the camera. For the sake of the example, please assume that it is fast enough / close enough that it is simply a blurred streak.

Even then, I believe what I said still holds - you're not actually recreating temporal detail; you're simply morphing objects into a spot that is inbetween the real frames.. If you wanted to more accurately recreate the detail, you'd need to perform some serious "un-motion-blurring" in addition to frame interpolation.

[morsa]

SoonUdie.
You are completely right, but I was just talking about a normal situation.Everybody knows a filter doesn't performs 100% OK with every situation, if this were the case what would you and I be doing here?
Anyway all of the really pro, really expensive software packages have the same limitations.That's why they ask the user to use high shutters.

[morsa]

I was wondering if I could use this approach to go from a 30p video clip to 60i to end with a 24p version of it.
I don't know how to set delta to get a double framerate from genmotion, so I could interleave the frames and obtain a 60i video.
Then I would use Scharfish's script to go from 60i to 24p.I'm sure this should work (although it won't be fast or easy) but I've got problems getting the correct params.My clip is from a JVC HD10U camcorder 1280x720.

[scharfis_brain]

@morsa. I've never used genmotion.
but why not going to 120fps and then doing a 1 out of 5 selection.
this would IMO be better than 60fps an then alternate blending (2x / 3x / 2x /3x ....)

[morsa]

Great idea!!
So now the question would be: How can I go from 30p to 120p with genmotion?

[scharfis_brain]

okay, after 10 minutes of testing, I've got running a framerate-doubler:

# script to create doubled frameratewith genmotion

loadplugin("C:\x\avisynth_c.dll")
loadCplugin("c:\x\genmotion.dll")
avisource("30p.avi")
i=converttoyv12()


f=GenMotion(i,filename="c:\deshaker.log",delta=0.5).trim(1,0)
b=GenMotion(i,filename="c:\deshaker.log",delta=-0.5)

# mix fwd. and bckwd. compensation
g=overlay(f,b,opacity=0.5) 

# make a 50:50 blend of the input
g1=overlay(i,i.trim(1,0),opacity=0.5) 

# mix the blended input into the compensated video to achieve some kind of stability
g2=overlay(g,g1,opacity=0.3) 

#double the framerate
interleave(i,g2)

#if the output of the interpolated frames is to bluury, try to reduce the 2nd opacity=0.3 value, or use
#interleave(i,g)

I'll post a quadrupler, when I am coming back from work (about 12hrs.)

and I think, we can nail static items in the video, so they aren't compemnsated, thus resultingin jittery station logos etc.
this would require introducing motionmasking with some tricks.

[scharfis_brain]

okay, the quadrupler has to wait.

For now, I've made a motioncompensator giving you a mostly clean output, avoiding swimming static parts of the video (TV-Station logos)
and avoiding false blending in differential motioned areas.
in both cases those affected areas are replaced with a blended clip of the neighboring frames.

this means:
a pan will be super-smooth.
a ball flying through the video while the panning occurs will have some kind of stutter, because it is NOT motioncompensated.

setmemorymax(256)
# script to create doubled framerate with genmotion
# while avoiding swimming static parts and false motion generated by genmotion

loadplugin("C:\x\masktools.dll")
loadplugin("C:\x\avisynth_c.dll")
loadCplugin("c:\x\genmotion.dll")

avisource("30p.avi")
i=converttoyv12()

th=7    #threshold for static detection
th1=7   #threshold of false-motion detection

#lower those thresholds for lesser repairing


#generate statice mask
i1=motionmask(i,thy1=th1,thy2=th1,thsd=255)


# make motion global compensation of the video
f=GenMotion(i,filename="c:\deshaker.log",delta=0.5).trim(1,0)
b=GenMotion(i,filename="c:\deshaker.log",delta=-0.5)

# motion compensate the static-parts-mask
f1=GenMotion(i1,filename="c:\deshaker.log",delta=0.5).trim(1,0)
b1=GenMotion(i1,filename="c:\deshaker.log",delta=-0.5)


# mix fwd. and bckwd. compensation
g=overlay(f,b,opacity=0.5) 

# mix the motioncompensated masks
g1=logic(f1,b1,mode="And")

#generate a falsemotionmask
g2=interleave(f,b).motionmask(thy1=th,thy2=th,thsd=255).selectodd().binarize().duplicateframe(0)

#merge static and false mask
g3=logic(g1,g2,mode="and") .blur(1).blur(1).binarize(threshold=1,upper=false)

# genrate blended clip out of the input to repair swimming static parts and false generated motion
h=overlay(i,i.trim(1,0),opacity=0.5)

# repair genmotion's output
j=overlay(h,g,mask=g3)


# for vis.
ih=interleave(i,h)
ig=interleave(i,g)
ij=interleave(i,j)
a=stackhorizontal(ih,ig)
b=stackhorizontal(ij,interleave(g1,g1))
stackvertical(a,b)

#stackhorizontal(g2,g1,g3)

#output the final clip
ij

my next ideas (after the quadrupler) will be:

NTSC <-> PAL conversion using this approach.
It will give super smooth pans without this annoying blending-motion on borders. while differnetial motion is converted via the standard blending method.
this will need a framerate 5 times as high as the input framerate for a NSTC -> PAL conversion.
and a framerate 6 times as high as the input framerate for a PAL -> NTSC conversion.

[morsa]

But I just wanted a converter from 30p to 24p
(just kidding)But in fact I just wanted that so the 120 fps approach should be fine for me, I guess......

[scharfis_brain]

Yes. You have to wait a little bit.
I have to restructurize my script, to do quad framerate.

this needs 3 times more motioncompensation!

[morsa]

Thank you, really

[scharfis_brain]

function mocomp(clip i, float amount, string file, int "th", int "th1",bool "vis")
	{
	th=default(th,7)
	th1=default(th1,7)	
	vis=default(vis,false)

	#generate static-area mask
	im=motionmask(i,thy1=th1,thy2=th1,thsd=255)

	# make motion global compensation of the video
	i2f=GenMotion(i,filename=file,delta=amount).trim(1,0)
	i2b=GenMotion(i,filename=file,delta=amount-1)

	# motion compensate the static-parts-mask
	i2fm=GenMotion(im,filename=file,delta=amount).trim(1,0)
	i2bm=GenMotion(im,filename=file,delta=amount-1)


	# mix fwd. and bckwd. compensation
	g2=overlay(i2f,i2b,opacity=amount)
	g2= vis ? g2.coloryuv(gain_u=48) : g2

	# mix the motioncompensated masks
	g2m=logic(i2fm,i2bm,mode="And")

	#generate a falsemotionmask
	g2mm=interleave(i2f,i2b).motionmask(thy1=th,thy2=th,thsd=255).selectodd().binarize().duplicateframe(0)

	#merge static and false mask
	g2mmm=logic(g2m,g2mm,mode="and").blur(1).blur(1).binarize(threshold=1,upper=false)

	# genrate blended clip out of the input to repair swimming static parts and false generated motion
	g2bl=overlay(i,i.trim(1,0),opacity=amount)
	g2bl= vis ? g2bl.coloryuv(gain_v=48) : g2bl

	# repair genmotion's output
	g2rep=overlay(g2bl,g2,mask=g2mmm)
	g2rep= vis ? g2rep.subtitle(string(amount)) : g2rep

	return g2rep
	}

this is the new motencompensation function

amount: float value between 0 and 1.
0 returns curr. frame
1 returns next frame
0.5 returns inbetween frame
0.25 the frame between 0 and 0.5
(and so on)

file: string to the location the deshaker.log (or whatever it is called)

th: int for false motion (differential motion) detection

th1: int for static area detection

vis: enable visualization
blue - motion compensated
red - blended (repaired areas)

usage:

general:

loadplugin("C:\x\masktools.dll")
loadplugin("C:\x\avisynth_c.dll")
loadCplugin("c:\x\genmotion.dll")

avisource("30p.avi")
i=converttoyv12()

possibilities:

framerate doubler:

i1=i
i2=i.mocomp(0.50,"c:\deshaker.log")
interleave(i1,i2)

framerate quadrupler:

i1=i
i2=i.mocomp(0.25,"c:\deshaker.log")
i3=i.mocomp(0.50,"c:\deshaker.log")
i4=i.mocomp(0.75,"c:\deshaker.log")

interleave(i1,i2,i3,i4)

30p to 24p conversion:

i1=i
i2=i.mocomp(0.25,"c:\deshaker.log")
i3=i.mocomp(0.50,"c:\deshaker.log")
i4=i.mocomp(0.75,"c:\deshaker.log")

interleave(i1,i2,i3,i4)

selectevery(5,1)

hope that helps.

[morsa]

Amazing results!!
I can't believe what I'm seeing came from 30p.Almost perfect to my eye
Although it has a little ghosting sometimes in sensitive areas as people heads with white hats more or less near the camera...

[scharfis_brain]

ghosting (or blending, to avoid stutter) is only present in areas with differntial motion.

but it is better live with little blending in those areas instead of haven frames full blended.

even a NTSC<->PAL conversion using this thingie is looking very smooth to me, no more annoying backward/forward motion on borders due to blending

theorethical:

NTSC to PAL

i1=i
i2=i.mocomp(0.2,"c:\deshaker.log")
i3=i.mocomp(0.4,"c:\deshaker.log")
i4=i.mocomp(0.6,"c:\deshaker.log")
i5=i.mocomp(0.8,"c:\deshaker.log")
interleave(i1,i2,i3,i4,i5)
changefps(50) #NO convertfps here! it will destroy the image

PAL to NTSC

i1=i
i2=i.mocomp(0.167,"c:\deshaker.log")
i3=i.mocomp(0.333,"c:\deshaker.log")
i4=i.mocomp(0.500,"c:\deshaker.log")
i5=i.mocomp(0.667,"c:\deshaker.log")
i6=i.mocomp(0.833,"c:\deshaker.log")

interleave(i1,i2,i3,i4,i5,i6)
changefps(59.94) #NO convertfps here! it will destroy the image

in practice, we first need to deinterlace the Video (maybe using motioncompensation using genmotion) and later reinterlace it.

I'll try to make a complete conversion fuction out of it.

[scharfis_brain]

@ fizick:
feature request:

could you add something like a forceprogressive-flag?

I want to fullframerate deinterlace a given video. and then do the motioncompensation.

with the current version of genmotion I only get the fields compensaten, not the whole frame.

this means, the number in the log file should be treated like progressive:

laced -> progressive
0B -> 0
1A -> 1
1B -> 2
2A -> 3
2B -> 4
3A -> 5
3B -> 6

would that be possible?

only this modification would it make possible to work without intermediate files when trying to do a standards conversion using genmotion.

[Fizick]

1. I very soon want release a new plugin for global motion compensation,
written from scratch, which will work without Deshaker.
This work take my time.
In have alpha version.

2. About force-progressive, i am not understand you:
Do you want compensate odd field to next odd,
and even to next even?