FrameRateConverter (Official)

[burfadel]

Originally Posted by MysteryX

I was looking for a way to fix duplicate frames and the solution was to replace them with frame interpolation, and I found FillDropsI from johnmeyer -- except that it uses basic interpolation without artifact masking. So I thought.. how about modifying it to use FrameRateConverter for interpolation?

Any comments on this?

I want to add "keep", 1 to keep 1st frame and interpolate the 2nd, and 2 to keep 2nd frame and interpolate the first, because in the video I'm testing the 2nd frame is better. I tried simply replacing the function YDifferenceFromPrevious with YDifferenceToNext but that didn't work. Any idea how to get that to work? Also is there a way to interpolate without separating fields? I might give better interpolation on whole frames, but I wasn't successful at doing that.

Once this script is polished I'll release it in the same script file.

function InterpolateDoubles(clip c, float "thr", string "preset")
{
    thr = default(thr, .1)
    preset = default(preset, "normal")

    even       = c.SeparateFields().SelectEven()
    even_flow  = FrameRateConverter(even, Preset=preset, FrameDouble=true).SelectOdd()

    odd        = c.SeparateFields().SelectOdd()
    odd_flow   = FrameRateConverter(odd,  Preset=preset, FrameDouble=true).SelectOdd()

    even_fixed = ConditionalFilter(even, even_flow, even, "YDifferenceFromPrevious()", "lessthan", string(thr))
    odd_fixed  = ConditionalFilter(odd,  odd_flow,  odd,  "YDifferenceFromPrevious()", "lessthan", string(thr))

    Interleave(even_fixed, odd_fixed)
    return Weave()
}

Running this script before mClean kind of works but it freezes (no MT)

There's been various scripts like that. There's blendupes:
https://forum.doom9.org/showpost.php...3&postcount=18

Morphdupes_MI
https://forum.doom9.org/showpost.php...7&postcount=20
https://forum.doom9.org/showpost.php...7&postcount=21
(split script)

Filldrops3
https://forum.doom9.org/showpost.php...92&postcount=1

Filldrops3 could very easily and effectively be incorporated . It would also have little impact on performance seeing as the mvanalyse has already been done.

[burfadel]

Did you find the solution? Using MFlowInter is a simple solution.

[MysteryX]

I haven't looked into it further. If I want to use MFlowInter, then I can't simply call FrameRateConverter and benefit from its features.

I'll also looking into increasing SkipOver from 120 to something like 210. Highly animated scenes get skipped and it breaks the smoothness.

Also using DCT=4 for MRecalculate is a good idea.

[burfadel]

Animated sources are difficult, ideally you need to incorporate using MCompensate, or MFlow and then use correction masks. MFlow does a great job most of the time, and makes up for it by adding it's own special effects at scene changes. Then you need the scene detection function... The good thing about using any of that is it can be used for normal scenes as well. Use normal clip for MAnalyse, use that for compensation, then use a super of that for MRecalculate. Or a variation on that principle.

[SpoCk0nd0pe]

From the latest changelog in mvTools2:

Fix: [DCT 8x8@8bit] safe multithreading for integer DCT (8x8 block size, 8 bit video): assembly had a single working buffer.

Does this fix the MT problems?

If I got it correctly, your block size recommendations are for 4:3. What blocksize would you chose for 1080p?

[MysteryX]

Originally Posted by SpoCk0nd0pe

From the latest changelog in mvTools2:

Fix: [DCT 8x8@8bit] safe multithreading for integer DCT (8x8 block size, 8 bit video): assembly had a single working buffer.

Does this fix the MT problems?

With BlkSize=8, CPU still goes only up to 29% with Prefetch(8)

Originally Posted by SpoCk0nd0pe

If I got it correctly, your block size recommendations are for 4:3. What blocksize would you chose for 1080p?

I treat 4:3 or widescreen videos the same.

As per documentation, default block sizes are, based on height
0-359: 8
360-749: 12
750-1199: 16
1200-1699: 24
1600-2160: 32

[MysteryX]

Version 1.1 is released!

What's new:
- Added InterpolateDoubles function to replace double frames with interpolated frames using FrameRateConverter
- Added DctRe parameter to specify DCT for MRecalculate
- DCT now also applied for MRecalculate by default
- Preset Normal now uses DCT=4 for MRecalculate. Old Normal is now Fast and old Fast is now Faster.
- Presets Slow and Slower also now use DCT=4 for MRecalculate
- SkipOver default treshold increased from 120 to 210
- BlendOver default treshold increased from 60 to 65
- SkipOver now specified for thSCD2 of MMask. Previously, scene changes would cause 2 adjacent frames to be skipped. Specifying thSCD2 causes only 1 frame to be marked for scene change.
- Updated Avisynth headers in DLL
- StripeMask: renamed trh parameter to thr

I also just did some tests with Stripes="Skip" and it's not practical for real videos, it often gives very weird results. I might remove it in next version.

I'm also open to feedback to improve InterpolateDoubles.

[MysteryX]

There is one issue with using DCT=4 for MRecalculate: it causes masks to be MUCH weaker, and thus pretty much disables artifact masking!

Would MFlowBlur be useful in that script? Would it be possible to use MFlowBlur instead of frame blending for artifact masking?

[burfadel]

It's possible, I was thinking that myself for mClean. For video temporal quality is much more important than single frame comparisons, particularly those where people have to zoom in or look hard to determine the difference. If there is fast motion it should be perceptively better. With motion analysis derived results, like an arm moving fast you get that opaque overlay look which is particularly noticeable in animation. MFlowBlur may make this area 'heavier' which wouldn't be good as it will make the arm look unnaturally fat momentarily. If you can detect that there is motion opaqueness caused by MFlowFPS, you can mask this and return the good detail from the previous or next frame. This will likely make the arm appear thinner. If you then apply MFlowBlur, you should get a smooth, and perceptively be motion artifact free assuming the source was free of this.

I'd say it would be difficult to implement! The temporal quality should be much higher though, and spatial comparisons should be more favourable as well.

[MysteryX]

How do you use MFlowBlur anyway? It doesn't have a FPS parameter like MFlowFPS, so how do I generate a matching clip?

[burfadel]

I believe it recreates existing frames based on the motion vector data of previous and next frames provided by MAnalyse. I was just thinking that if you ran MFlowBlur or Mcompensate before MflowFPS, and work out the clip stuff (super,superfilt) based on this, it could produce good results? I think it all comes down to experimenting with the functions.

[MysteryX]

I asked Chainik about it

Originally Posted by Chainik

> Does MFlowBlur generate artifacts?

yes, since it uses the very same (i.e. wrong) motion vectors

It's useless then.

I also tried it. Takes the source clip and kind of blurs/distorts it. It's not generating any in-between frames. Useless.

[MysteryX]

OK after testing the mask of MRecalculate with DCT=4, overall, MAnalyze gives a much weaker mask with DCT=4 than with DCT=0. DCT=4 and DCT=1 give the same mask. If you do MAnalyze with DCT=0 and MRecalculate with DCT=4, you get the same mask as if MAnalyze was done with DCT=4.

Thus, with the latest release, presets Slow/Slower that were already using DCT=4 will behave similarly but with slightly better quality. Preset Normal will have much lower artifact masking because of MRecalculate. For artifact masking, perhaps some adjustment could be made to take this mask strength discrepancy into consideration.

This however can cause issues for scene change detection if masks are too weak.

In terms of mask strength discrepancy, here's what I'm talking about

MAnalyze DCT 0, 1, 2, 3, 4


MAnalyze+MRecalculate DCT 0, 1, 2, 3, 4

[hello_hello]

MysteryX,
I was experimenting with frame interpolation on some animation today and noticed FrameRateConverter (both versions 1.0 and 1.1) tends to lose the plot during fade-outs. I haven't fiddled with settings too much yet so I don't know if there's an adjustment that'd fix it, although it seems to be the sort of thing that probably shouldn't happen by default.

There's three MKVs in the zip file I've linked to if you'd care to look. "12fps.mkv" is simply a section of the original DVD cropped and resized and decimated to 12fps. It's for reference.
"FrameRateConverter.mkv" is the result of adding FrameRateConverter(24,1) to the script after resizing, and "Interframe.mkv" is the result of adding Interframe(cores=1, NewNum=24, NewDen=1).

FrameRateConverter seems to do a better job than Interframe aside from the fade-outs. If you look at the grey area near the middle of the picture a bit after frame 120 and step through the frames one at a time as it fades out, you'll see the shadow over the grey area appears to move up and down. A similar thing happens a little later as the girl fades out and the shot of the earth behind her distorts a little in the direction it's moving. It appears FrameRateConverter might be mistaking the fading-out for movement.

Cheers.

12fps-24fps.zip (3.3MB)

Edit: For the record I tried manolito's earlier version of the script here and the fade-out problem was far worse. I must have saved StainlessS's DoubleRate script at some stage so I tried it and the effect was reduced, although still present.

Thanks.

[MysteryX]

I've now adjusted the masks to look similar with different DCT values. Still need to do some testing.

hello_hello, for anime, use Preset="slow". With DCT=4, the fade out is good. There's a bit of a warping effect at 239-242. When the lady fades out, the mask is too weak to skip that, but on fade in, I get a Skip value of 46. Set BlendOver=40. I think it's a good idea to set BlendOver to a lower value for anime.

Perhaps I could add a preset "Anime" that changes BlendOver/SkipOver values, but that would require more testing. I'll release this version with re-adjustments to the masks and we'll see from there what settings need to be tweaked for anime.

Also, if you double the frame rate, use FrameDouble=true instead of NewNum to preserve source frames. I'm editing the script to set FrameDouble to true if NewNum is double the frame rate.

[MysteryX]

Version 1.2 is ready!

What's new:
- Mask strength now adjusted to produce similar artifacts masking with different DCT values
- Removed Stripes parameter as Skip isn't practical
- Added Stp parameter, whether to detect and blend stripes
- Changed MaskTrh default from 100 to 120
- Changed BlendOver default from 65 to 70
- If NewNum is twice the clip's frame rate, set FrameDouble to true by default to preserve source frames

You can test preset="anime" to see which settings work best. This new preset hasn't yet been tested and is likely to be tweaked.

[manolito]

Originally Posted by hello_hello

Edit: For the record I tried manolito's earlier version of the script here and the fade-out problem was far worse.

The only difference with your command line between FRC 1.0 and my simplified script is the block size. FRC defaults to 12 for this frame size while my script uses 16 (a block size of 12 is not supported by older versions of MVTools2).

I did a few tests, and for me the clip looks best with a block size of 32 (happens often with anime). DCT=1 did not make much difference, though. Using blksize=32 fixes the fluttering grey shadow completely, but when the planet enters the window, the first couple of frames have distorted movement. I think that this is far less visible than the grey shadow, it only can be detected by stepping through the frames.


Cheers
manolito

[burfadel]

Originally Posted by MysteryX

Version 1.2 is ready!

What's new:
- Mask strength now adjusted to produce similar artifacts masking with different DCT values
- Removed Stripes parameter as Skip isn't practical
- Added Stp parameter, whether to detect and blend stripes
- Changed MaskTrh default from 100 to 120
- Changed BlendOver default from 65 to 70
- If NewNum is twice the clip's frame rate, set FrameDouble to true by default to preserve source frames

You can test preset="anime" to see which settings work best. This new preset hasn't yet been tested and is likely to be tweaked.

You didn't make a binary of this version, you only provided the source code .

[hello_hello]

Thanks for the info MysteryX and manolito.

I haven't had a chance to play around more yet. Probably not until tomorrow, but I was wondering.....

I didn't give any thought to it at the time, but would frame interpolation work better at higher resolutions? I resized first, thinking it'd be faster:

TDecimate(mode=2, rate=12, Input="D:\ep1.txt")
crop(14, 4, -14, -4)
Spline36Resize(704,396)
FrameRateConverter(24,1)

I'll use FrameDouble=true next time, but would it make much difference if I put FrameRateConverter before the cropping and resizing? It's a PAL DVD so there's 720x576 resolution to begin with, even if there's not 720x576 worth of picture detail as such.

Cheers.

[MysteryX]

Originally Posted by manolito

I did a few tests, and for me the clip looks best with a block size of 32 (happens often with anime).

Right, forgot about that. Preset "anime" should use higher block sizes by default, although that still needs to be tested and tweaked.

Originally Posted by burfadel

You didn't make a binary of this version, you only provided the source code .

Oups forgot to upload! Fixed

Originally Posted by hello_hello

would it make much difference if I put FrameRateConverter before the cropping and resizing?

For 288p to 768p, it's much better if interpolation is done last. For 720p to 1080p, I'm not sure whether it's best to interpolate before or after, I still need to test that. One thing for sure: interpolation is expensive. If you interpolate after upscaling, it will be slower. Test both ways, opening each version in a different instance of VirtualDub, and compare the frames.

I've just re-encoded a very difficult and animated video with this version. Quality is considerably improved. Output file size is also considerably different: 209mb instead of 205mb.