Blend duplicate frames?

[bcn_246]

Hey, I have some footage with quite a few duplicate frames. I have found a number of scripts that use interpolation (FillDrops, MorphDups, DoubleDropFix, GameDropFix) however a simple blend (like that used in ConvertFPS) would be preferable. Does such a script exist, and if not, would it be possible to modify something like the following to use a simple blend rather than interpolation with MVTools/SVP?

TIA

# filldrops3 based on MugFunky's filldrops from 19.December 2005, extended by Emulgator on 26.Feb.2012 to accommodate interpolation of Doupledrops
# http://forum.doom9.org/showthread.php?p=753779#post753779
# "...so i wrote a little function to replace all frames that are identical to the previous one with a motion-interpolation of the adjacent frames.
# it could be useful for captures that you can't get again, but are full of drops..."
# Needs MVTools2, deviation now tunable.
function filldrops3 (clip c, float "dev")
{
blksz=16 # Blocksize for Motion search. Default = 8. My Default 16: Delivers a smoother picture. 4: Gives too many artefacts with poorer source.
searchp=32 # Radius for Motion search. Default = 2, My Default = 32
bld=true # Decides what to do if a Scenechange is detected by the following parameters: true will blend, false will repeat previous frame.
SCDpb=800 # Default = 400. thSCD1. SAD (pixels*luma values) Threshold per Block. A Block exceeding this SAD is considered changed.
##                     SAD = Sum of Absolute Differences  = Number of pixels per 8x8 block (=64) multiplied with their luma difference.
##                     SCDpb is always given related to 8x8 blocks and scaled internally to match other block sizes. SCDbp is thSCD1 and is evaluated before SCDpf=thSCD2
SCDpf=200 # 0-255, Default=130. thSCD2. Changed Blocks percentage threshold per frame. 0=0%, 127=50%, 255 = 100%.
##                     This Threshold sets which minimal percentage of changed blocks a frame has to contain to be considered as a scene change.

dev=default(dev, 2.0)
super=MSuper(c,pel=2)

back_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=true, delta=1, temporal=false, dct=0, divide=0, trymany=false)#delta=framedistance;isb=IS Backward
		# Looks 1 frame ahead and calculates vectors backward to reference frame.
forw_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=false, delta=1, temporal=false, dct=0, divide=0, trymany=false)#search 3:=Exhaustive; 4:=Hex(default); 5:=UMH
		# Looks 1 frame behind and calculates vectors forward to reference frame.
dblback_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=true, delta=2, temporal=false, dct=0, divide=0, trymany=false)#delta=framedistance;isb=IS Backward
		# Looks 2 frames ahead and calculates vectors backward to reference frame. To be used when reference frame + 1 has to be excluded.
dblforw_vec=MAnalyse(super, blksize=blksz, search=5, searchparam=searchp, isb=false, delta=2, temporal=false, dct=0, divide=0, trymany=false)#search 3:=Exhaustive; 4:=Hex(default); 5:=UMH
		# Looks 2 frames behind and calculates vectors forward to reference frame. To be used when reference frame -1 has to be excluded.

global fillsingdrops = MFlowInter(c,super,back_vec,forw_vec,time=50, blend=bld, thSCD1=SCDpb, thSCD2=SCDpf)
global filldoubdrops33 = MFlowInter(c,super,dblback_vec,dblforw_vec,time=33, blend=bld, thSCD1=SCDpb, thSCD2=SCDpf)
global filldoubdrops67 = MFlowInter(c,super,dblback_vec,dblforw_vec,time=67, blend=bld, thSCD1=SCDpb, thSCD2=SCDpf)
global oric=c
global devthresh=dev


#ConditionalFilter(c, c, fillsingdrops, "YDifferenceFromPrevious()", ">", "devthresh", show=true)##works for mending of single framerepeats
#ConditionalFilter(c, c, filldoubdrops33, "YDifferenceFromPrevious()>devthresh", "&&", "YDifferenceToNext()>devthresh", show=true)##wrong syntax

c.scriptclip("""(YDifferenceFromPrevious<devthresh) && (YDifferenceToNext<devthresh) ? (filldoubdrops33++filldoubdrops67.Trim(1,0)) : (YDifferenceFromPrevious<devthresh) ? fillsingdrops : oric""")

}

[LoRd_MuldeR]

Doesn't Dup with "blend=true" do exactly that?
http://rationalqm.us/dup/dupnew.html

[bcn_246]

Originally Posted by LoRd_MuldeR

Doesn't Dup with "blend=true" do exactly that?
http://rationalqm.us/dup/dupnew.html

It does... kind of... however with very poor results (the threshold has to be set high to detect duplicates, and ends up blending a lot of good frames also). I have tried de-noising prior to running Dup and things don't get much better. The aforementioned interpolation scripts, while not perfect, do a much better job of detecting what to interpolate. The issue is with the artefacts...

The script posted seems to do the best job when it comes to detecting duplicates, just the artefacts are way to much (thus a simple blend, like that used in Dup, would be much preferred).

[johnmeyer]

I fully understand that motion estimation sometimes creates artifacts. Here is a version of filldrops that simply blends the current frame with the previous frame:

source=AVISource("e:\fs.avi").convertTOYV12()
merged=filldrops(source)
#stackhorizontal(source,merged)
return merged

function filldrops (clip c)
{
  prev=trim(c,1,0)
  global filldrops=merge(c,prev)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
  return fixed
}

I hardwired the threshold to "0.1". If that detects too few or too many duplicates, then change it.

Also, the simple motion-estimated version of filldrops() that I created a few years ago does a pretty good job with many clips. This is the same as MugFunky's original code for progressive video, but modified to use MVTools2 instead of the original MVTools. Feel free to try this out, if you don't like the ghosting that is inherent in blending adjacent frames (as the above code does):

function filldrops (clip c)
{
  super=MSuper(c,pel=2)
  vfe=manalyse(super,truemotion=true,isb=false,delta=1)
  vbe=manalyse(super,truemotion=true,isb=true,delta=1)
  filldrops = mflowinter(c,super,vbe,vfe,time=50)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
  return fixed
}

[bcn_246]

Originally Posted by johnmeyer

I fully understand that motion estimation sometimes creates artifacts. Here is a version of filldrops that simply blends the current frame with the previous frame:

source=AVISource("e:\fs.avi").convertTOYV12()
merged=filldrops(source)
#stackhorizontal(source,merged)
return merged

function filldrops (clip c)
{
  prev=trim(c,1,0)
  global filldrops=merge(c,prev)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
  return fixed
}

I hardwired the threshold to "0.1". If that detects too few or too many duplicates, then change it.

Also, the simple motion-estimated version of filldrops() that I created a few years ago does a pretty good job with many clips. This is the same as MugFunky's original code for progressive video, but modified to use MVTools2 instead of the original MVTools. Feel free to try this out, if you don't like the ghosting that is inherent in blending adjacent frames (as the above code does):

function filldrops (clip c)
{
  super=MSuper(c,pel=2)
  vfe=manalyse(super,truemotion=true,isb=false,delta=1)
  vbe=manalyse(super,truemotion=true,isb=true,delta=1)
  filldrops = mflowinter(c,super,vbe,vfe,time=50)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
  return fixed
}

Thanks!

Would it be hard to create a simple threshold variable for the function (rather than editing the .avsi)?

I tried

function filldrops (clip c, float "dev")
{

dev=default(dev, 0.1)

  prev=trim(c,1,0)
  global filldrops=merge(c,prev)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "dev")
  return fixed
}

But threw up an error, I think I am pretty close though...

[Gavino]

Originally Posted by bcn_246

I tried

function filldrops (clip c, float "dev")
{

dev=default(dev, 0.1)

  prev=trim(c,1,0)
  global filldrops=merge(c,prev)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "dev")
  return fixed
}

But threw up an error, I think I am pretty close though...

Try changing it to this:

  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", string(dev))

I don't think 'filldrops' needs to be global either, as it's not referenced by name at run-time.

[bcn_246]

Thanks

Is there any way of implementing a set of rules using the ConditionalFilter that prevent it from blending at scene change points (like that found in filldrops3)?

Thanks a bunch for all your help!

[StainlessS]

Here something, not much tested.

Code REMOVED See post #20.

[johnmeyer]

Originally Posted by Gavino

I don't think 'filldrops' needs to be global either, as it's not referenced by name at run-time.

I know you don't like me using globals, but it most definitely did not work until I added it (the variable cannot be used inside a conditional without it).

I did try, but it didn't work until I added Global.

P.S. for StainlessS

I only tested on video that had a single duplicate at a time (i.e., I did not test it on three duplicates in a row). It worked exactly as expected, although I only did enough testing to make sure that it passed through original frames except when a duplicate was detected, and that the replaced frame was a 50/50 merge of the duplicate and the previous frame.

[johnmeyer]

Originally Posted by bcn_246

Is there any way of implementing a set of rules using the ConditionalFilter that prevent it from blending at scene change points (like that found in filldrops3)?

Probably, but it's a lot more work, especially since scene detection logic screws up a lot.

[StainlessS]

John you misunderstand, Blended frame should come from those either side of dupe, see edit to prev post.
EDIT: Although I guess if curr is dupe of prev then using curr is not so different.
But Prev variable name in quoted code snippit is at least misleading (should read Next).

EDIT:

especially since scene detection logic screws up a lot

The Th2 arg in BlendDupes() is not so much scenechange detection, but detecting whether blend produced would look bad,
but to a certain extent that would also include scenechanges. If blend would look bad, then duplicate will not be replaced with blend.

[StainlessS]

Oops bug fix in post #8

    Th  = Float(Default(Th,0.5))
    Th2 = Float(Default(Th,8.0))  # Th should have read Th2

Few other small mods.

[johnmeyer]

Originally Posted by StainlessS

John you misunderstand, Blended frame should come from those either side of dupe, see edit to prev post.

I saw your post. I did actually run the script before I posted, and when I viewed the original and the modified version side-by-side, the script did put the blended frame at the correct spatial location (at least that's how it appeared to me). So I think it does what the OP asked for.

[StainlessS]

John, yes but the blended frame is a blend of current and Next frame, if that is what you intended, I was suggesting that variable name is misleading as named
Prev rather than Next and also questioning whether should be using Prev and Next rather than Curr and Next. (but if Curr is dupe of Prev then not much different).
Trim(1,0) shifts the clip Next (not Prev) frame to align with current.

Post #8 update, Added MinUniq. Allows restriction on consecutive blends replaced.

EDIT: Post #8 Updated. Removed MinUniq arg, Changed Th default to 0.25. Change in detection logic. Assigned v1.0.

[johnmeyer]

Originally Posted by StainlessS

John, yes but the blended frame is a blend of current and Next frame, if that is what you intended, I was suggesting that variable name is misleading as named
Prev rather than Next and also questioning whether should be using Prev and Next rather than Curr and Next. (but if Curr is dupe of Prev then not much different).
Trim(1,0) shifts the clip Next (not Prev) frame to align with current,

I copy/pasted from several scripts and put this together in one minute. I didn't even look at the variable names. Sorry about the prev/next confusion. Hey, doesn't everything in AVISynth work backwards (start at end of script and work backwards)?

OK, lame excuse for sloppy work ...

,,, but speaking of "work," it does do what the OP asked for ... I think ...

[Gavino]

Originally Posted by johnmeyer

I know you don't like me using globals, but it most definitely did not work until I added it (the variable cannot be used inside a conditional without it).

I did try, but it didn't work until I added Global.

Neither of the functions in post #4 use the variable 'filldrops' inside the conditional (ie in the string arguments, which are evaluated at run-time). In both functions, the variable is simply passed as a normal argument to ConditionalFilter, which is no problem. (And in the second function, it is not declared 'global'.)

[StainlessS]

Sorry John, I was not getting at you, just being a bit pedantic I guess, call your variables eg 'MoonbaseAlpha' if you like.

[bcn_246]

Thanks StanlessS... I tired using the earlier version of your script (reposted below) and got really great results. With "MinUniq = 0" and "Th2 = 16" it did exactly what I had in mind.

Can you expand a bit on what exactly the "Th" variable does? It is "Th2" that seems to be the biggest modifier for detection sensitivity. Also can I ask why you removed the MinUniq variable? On my source I had some blocks of 2 or 3 dropped frames.

Again, really appreciate all the time you guys have given

(Below is the version I had good results from)

Function BlendDupes(clip c,Float  "Th",Float "Th2",Int "MinUniq",bool "Show") {
# BlendDupes() by StainlessS : http://forum.doom9.org/showthread.php?p=1764219#post1764219
# PROGRESSIVE. Req RT_Stats, Grunt.
# Replace dupes with a frame blended from those either side.
# Th,  Default 0.5, YDifference Prev->Curr <= Th  Then current frame is Dupe.
# Th2, Default 8.0, YDifference Prev->Next <= Th2 Then dif between frames either side of current is not so far 
#      apart that would produce bad blend (eg not scene change).
# MinUniq default 1, Minimum number of Unique frames between dupes replaced with blends.  0 or more.
#   0 = Replace ALL dupes even consecutive ones.
#   1 = consecutive dupes not replaced, at least 1 original frame left between blended replacements.
#   Etc.  
# Show: Puts indicator on fixed frames.
    c
    Th      = Float(Default(Th,0.5))
    Th2     = Float(Default(Th2,8.0))
    MinUniq = Default(MinUniq,1)
    Show=Default(Show,False)
    Assert(MinUniq>=0,"BlendDupes: 0 <= MinUniq")
    Function BlendDupes_lo(clip c,clip BlendC,int n,Float Th, Float Th2,Int MinUniq,Bool Show) {
        c
        dif_p  = RT_YDifference(delta=-1)         # Diff Prev <-> Curr
        dif_pn = RT_YDifference(n-1,delta=2)      # Diff Prev <-> Next (either side of current)    
        Blend  = (n-GLB_BlendDupes_LastBlend>MinUniq && dif_p <= Th && dif_pn <= Th2)
        Last = (Blend) ? BlendC : Last
        (Blend) ? RT_IncrGlobal("GLB_BlendDupes_BlendCount") : NOP
        (Show) ? RT_subtitle("%d] Blends=%d : LastBlend=%d : Uniq=%d\n   Prev->Curr : Prev->Next : Blend\n   %7.3f      %7.3f      %s",
               \ n,GLB_BlendDupes_BlendCount,GLB_BlendDupes_LastBlend,n-GLB_BlendDupes_LastBlend-1,dif_p,dif_pn,Blend) : NOP 
        (Show) ? RT_Subtitle("Th=%.3f : Th2=%.3f : MinUniq=%d",Th,Th2,MinUniq,Align=1) : NOP
        Global GLB_BlendDupes_LastBlend = (Blend) ? n : GLB_BlendDupes_LastBlend
        Return Last    
    }
    PrevC  = DeleteFrame(FrameCount-1).DuplicateFrame(0)        # Or SelectEvery(1,-1), I but prefer identical length clips ALWAYS.    
    NextC  = DuplicateFrame(FrameCount-1).DeleteFrame(0)        # Or SelectEvery(1,1)    
    BlendC = Merge(PrevC,NextC)
    BlendC = (Show) ? BlendC.Subtitle("FRAME FIXED",size=30,text_color=$0000FF,align=5) : BlendC
    Global GLB_BlendDupes_BlendCount = 0    Global GLB_BlendDupes_LastBlend = 0
    Return Last.Scriptclip("BlendDupes_lo(Last,BlendC,current_frame,Th,Th2,MinUniq,Show)",Args="BlendC,Th,Th2,MinUniq,Show")
}

[StainlessS]

I deliberately changed function to cater for low motion scenes (ie not replace low motion), unfortunately it is intended that it should be for
single dupe detection only (what I would mostly have to deal with). I guess I could put back MinUniq functionality, somehow, when I figure out
what the rules are

What am I supposed to use for the blends where multiple consecutive dupes exist ?

[StainlessS]

Here, MorphDupes_MI.avs, Should fit the bill nicely.

EDIT: Previous scripts were without having a test clip, capped some webcam footage at night and because of low light conditions
had plenty of dupes, so this script tested and working very well, I think.

MorphDupes_MI.avs Part 1 of 2 (Too big, Copy/Paste together)

Function MorphDupes_MI(clip c, Int "DifbyN",Float "Th",Int "DifbyN2",Float "Th2",Float "CW",Int "MaxInterp",
          \ Bool "Stack", bool "Show", Bool "ForceProcess",Bool "Verbose",
          \ int "pel",int "sharp",int "rfilter",Float "ml",Int "SC_thSCD1",Int "SC_thSCD2") {

/*
    MorphDupes_MI, by StainlessS. Req MvTools (c) Manoa/Fizick & Others : GScript and Grunt (c) Gavino : RT_Stats, TWriteAVI v2.0, CallCmd, (c) StainlessS. 
    http://forum.doom9.org/showthread.php?p=1764865#post1764865

    Finds runs of duplicates and replaces them with either Interpolated or blended frames.
    
    Planar, YUY2. Audio, no change. Multi-Instance capable.
    
    MorphDupes_MI(clip c, Int "DifbyN"=-1,Float "Th"=0.0,Int "DifbyN2"=-1,Float "Th2"=16.0,Float "CW"=1.0/3,Int "MaxInterp"=3,
          \ Int "Stack"=False, bool "Show"=False,Bool "ForceProcess"=False,Bool "Verbose"=False,
          \ int "pel"=2,int "sharp"=2,int "rfilter"=2,Float "ml"=100.0,Int "SC_thSCD1"=400,Int "SC_thSCD2"=130)
    
        There are two Difference measuring options to measure whether a frame is a duplicate or not and switched via DifByN, if DifByN is -1 (default),
    then uses RT_FrameDifference function (together with CW=ChromaWeight setting, default 1.0/3, 0.0=Luma only) and Th threshold to determine
    duplicate status. RT_FrameDifference returns the average pixel difference for the two comparison frames.     
    If DifByN is >= 0, then switches to using RT_LumaPixelsDifferent(Thresh=DifByN) where DifByN is also a control arg to
    RT_LumaPixelsDifferent. RT_LumaPixelsDifferent returns the percentage of pixels where the luma difference between corresponding pixels is
    greater than a threshold (DifByN). If the difference measure between current frame and it's predecessor (measured via RT_Framedifference or 
    RT_LumaPixelsDifferent) is <= Th, then it is deemed a duplicate frame.   
    Elsewhere we use the term 'Unique Start' frame, this is the frame at head of run of dupes [the frame of which others are a copy], and unique in 
    that it is different from its predecessor.

    Frames Determined to be Duplicates:
        For frames that are determined to be duplicates, there are two modes of operation, Interpolation, and Blend, the two modes are partly user
        adjustable and partly automatic. MaxInterp (default 3) governs the greatest number of duplicate frames that can be Interpolated, setting it to 0 
        will be always in Blend mode. If a Scene change is detected via mvtools MSCDetection(thSCD1=SC_thSCD1,thSCD2=SC_thSCD2) on the last frame in
        duplicate run, and the frame following a run of duplicates, then it will also drop down to Blend 
        mode, using the Start Unique frame and frame BEFORE End Unique frame as the source frames of blend. In addition, if the End Unique frame is NOT 
        a scene change, but it is too different to the Start Unique frame (via DifByN2 and Th2), then it will again drop to blend mode but using the 
        End Unique frame as a blend source frame (DifByN2 and Th2 work in exactly the same way as previously noted for DifByN and Th).
    
    Args:-
        DifByN, Default -1. -1 or greater. Frame comparison mode selection and thresh arg to RT_LumaPixelsDifferent() function (when DifByN >= 0).
        Th, Default 0.0, A difference Threshold for value returned by either RT_FrameDifference or RT_LumaPixelsDifferent() to determine duplicate
            status of a frame.
              DifByN == -1  : Uses RT_FrameDifference as measure.
              DifByN >= 0   : Uses RT_LumaPixelsDifferent as measure.
            DifByN and Th are used to establish duplicate status of a frame, if result of above DifByN selected function is <= Th, then frames are 
            judged to be same (dupe).
            By default it looks for Indentical frame dupes. 

        DifByN2, Default -1. -1 or greater. Frame comparison mode selection and thresh arg to RT_LumaPixelsDifferent() function (when DifByN2 >= 0).
        Th2, Default 16.0, 0.0 or more. Th2 MUST be greater than Th if DifByN2 comparison mode selection is the same as DifByN.  
            DifByN2 and Th2 work in exactly the same way as DifByN and Th.            
            Th2 is a difference Threshold for value returned by either RT_FrameDifference or RT_LumaPixelsDifferent() to determine
            if the duplicate run Unique Start frame is too different to the Unique End frame (that follows a run of duplicates).
            If the Unique End frame IS too different to the Unique Start frame, then switches down to blend mode for the current run of dupes.
        
        CW, Default 1.0/3.0, 0.0->1.0. ChromaWeight arg to RT_FrameDiffernce (if DifbyN or DifByN2 are -1), 0.0 = Luma only.
        
        MaxInterp, Default 3, 0 -> 9. Maximum number of frames that may be Interpolated.
                
        Stack, default False, If True show StackVertical the difference between current frame and its predecessor. Switched off if Show is false.

        Show, Default False. Show some metrics stuff if true. Switched off if ForceProcess = true. 

        ForceProcess, Default false. If true then scan entire clip doing the dupe blend replacement automatically, and then return result when
            completed. Will take time to scan entire clip. Auomatically switches OFF Show and Stack when ForceProcess = true.

        Verbose, Default False. Show some args if true and Show=True. 

        pel,     Default 2,     MvTools arg to MSuper() for interpolation. See MvTools. range=1 or 2 or 4 
        sharp,   Default 2,     MvTools arg to MSuper() for interpolation. See MvTools. range=0 -> 2
        rfilter, Default 2,     MvTools arg to MSuper() for interpolation. See MvTools. range=0 -> 4
        ml       Default 100.0, MvTools arg to MFlowInter() for interpolation. See MvTools. range=greater than 0.0.
        SC_thSCD1 Default 400,  MvTools arg to MSCDetection(thSCD1) for scene change detection. See MvTools.range = 0->(8*8)*255
        SC_thSCD2 Default 130,  MvTools arg to MSCDetection(thSCD2) for scene change detection. See MvTools.range = 0-> 255

    ######
   
    Metrics shown:- 
      nnnn] UDSIB {a/b} : RunCount=c : RunMax=d
      Prev->Curr = fff.ffffff
      FrameDif   = fff.ffffff (Unique<->n,      DifByN)
      RunDif     = fff.ffffff (Unique<->Unique, DifByN2)
      BlendWeight= fff.ffffff
      
      Where, 
          nnnn, = Frame Number
          Flags UDSIB (where hi-lited)
                  U = Unique frame (not at duplicate)
                  D = Duplicate.
                  S = Scene Change follows duplicate run.
                  I = Interpolate mode used.
                  B = Blend mode used.
          {a/b}     = a'th frame in run of b dupes.
          RunCount=c, = Number of duplicate runs encountered so far.
          RunMax  =d, = length of longest duplicate run encountered so far.
          Prev->Curr  = DifByN metric difference between current frame and its predecessor.
          FrameDif    = DifByN metric difference between Unique Start frame and current duplicate frame. (Only shown for Duplicates)
          RunDif      = DifByN2 metric difference between Unique Start frame and Unique End frame following duplicate run. (Only shown for Duplicates)
          BlendWeight = Weight arg to Merge() two frames in blend mode. (Only shown for Blended Duplicates)
           
*/

EDIT: Added stuff in BLUE
EDIT: I'm not sure how useful the DifByN>=0 (not the DifByN2) mode will be, consider it experimental, no harm to leave it
as it is. DifByN2 mode though is possibly of some use, eg DifByN2=8, Th=50.0, would switch down to Blend mode
when 1/2 of End Unique frame pixels are different to the Start Unique frame by 8 or more luma levels.