FrameRateConverter (Official)

[MysteryX]

I'm looking at Dogway's version. Output="diff" shows nothing. Masks are very different than they should be and the output is completely different. Output hasn't been compared with the base script for equivalence.

v2.0 is nearly working.

Quick question. How do I access the clip's _Matrix property (and format) to convert it back and forth correctly?

video = video.resize.Bicubic(format=vs.RGBS)
video = video.rife.RIFE(uhd=False)
video = video.resize.Bicubic(format=vs.YUV420P8, matrix_s="709")

and...... RIFE doesn't have any parameter to specify the frame rate??

[MysteryX]

Beta 9. Looks functional!

Test it and let me know if you find problems. It contains both VapourSynth and Avisynth scripts.

TODO:
- implement RIFE interpolation with format conversion to RGBS back-and-forth, then you can call FrameRateConverter(rife=True)
- add fullRange parameter in the Avisynth version.

[Dogway]

I'm waiting on 2.0 so I can finish my mix mod.
I normally compare the outputs to the T so they match, and on any bitdepth, this has been so with ExTools and other mods.
As for FrameRateConverter there were a few odd things:

Was this really necessary?

.mt_expand(mode= mt_circle(zero=true, radius=8))

This is utterly slow, even in ExTools is slow so I used an approximation, so yes is not mathematically exact, if it was me personally I would resize down (further) and use rad=4.

Another thing, FRC_GaussianBlur42(2.8).
Again the concept is good but the values represent nothing, you are blurring on the downscaled clip and hence breaking the blur linearity.
I made ex_GaussBlur() based on this, removed the explicit blur() and parametrized rad to respond to an approximated gaussian standard deviation, so again not mathematically exact to your output, but things make more sense. If you want to be mathematically accurate to a gaussian use vsTCanny() in mode=-1, but it will never match FRC_GaussianBlur42().

Other than that I don't recall changing anything else.

[poisondeathray]

Originally Posted by MysteryX

How do I access the clip's _Matrix property (and format) to convert it back and forth correctly?

format=clip.format

Not sure about matrix, but doesn't really matter, because you're using the same one both ways . So if you randomly choose "709" it "reverses" when you go back and forth. Also if input clip is "unspec" or unassigned, it might cause problem so just picking one should work

RIFE doesn't have any parameter to specify the frame rate??

RIFE only does powers of 2, you need to also use mvtools2 or other for intermediate rates

There was talk on the official implementation page of eventually including other rates, but it hasn't materialized yet

[MysteryX]

Dogway, I'm getting entirely different mask output -- doesn't even compare. Wrong dependency perhaps?

FrameRateConverter(60, preset="slower", blkSize=16, output="auto", debug=True)

Using
FrameRateConverter-Dogway 1.4.2 (28-06-2021)
ExTools v2.7 (01-07-2021)
Utils-r41 v0.41 (2017-05-13)

Originally Posted by poisondeathray

RIFE only does powers of 2, you need to also use mvtools2 or other for intermediate rates

There was talk on the official implementation page of eventually including other rates, but it hasn't materialized yet

...

Well that's a "slightly" limiting factor.

Still, could very well double the frame rate before using the exact same frame blending.

[kedautinh12]

Why you don't test new ver??
Frame Rate Converter mix mod Version: 1.5.0 (06-09-2021)
ExTools v5.8 (18-09-2021)

[poisondeathray]

Originally Posted by poisondeathray

RIFE only does powers of 2, you need to also use mvtools2 or other for intermediate rates

There was talk on the official implementation page of eventually including other rates, but it hasn't materialized yet

Originally Posted by MysteryX

Well that's a "slightly" limiting factor.

Still, could very well double the frame rate before using the exact same frame blending.

It's still beneficial in scenes where RIFE works well. I posted some examples, such as 23.976=> 47.952 RIFE => 59.94 MVTools2. The "distance" that mvtools2 has to interpolate in the 2nd step is smaller, and the end result has minimal/negligible artifacts. But if you use MVtools2 directly from 23.976 => 59.94, there are the usual artifacts

Having the option to use RIFE I think is still a plus overall. Also consider implementing a switch for model version (anime model, vs. 2.4, vs. 3.8 etc...)

[MysteryX]

Beta 10.

The moment you've all been waiting for: RIFE support. Set rife=1 to run it once, or rife=2 to run it twice on the frame blending clip used for artefact masking. This will reduce frame blending in artefact areas.

It however doesn't have as huge of an impact as one might think, but definitely helps in some frames. You can also lower the mask threshold to see more of the RIFE clip.

First running RIFE and then running FrameRateConverter on top of it would also be an option.

btw I'm unable to set gpu_id, how do I set it to my 2nd NVIDIA graphic card? Haven't found how to use list_gpu param either.

TODO: when preset="anime", set model=2

btw I had to comment this validation in the source code to avoid an error when doubling framerate twice. Any idea what this validation was for? That was copy/pasted from "somewhere" LOL (isn't that how doom9 works?)

[MysteryX]

Playing with it, couple of issues.

When I use RIFE in 2 separate VirtualDub windows, sometimes junk comes through (in artefact areas).

As for difficult frames marked as full-frame blending, RIFE isn't doing any magic so it's not giving a good output.

More tweaking will be required.

[MysteryX]

Good news. After keeping full frame blending on difficult frames, and simply adding Maximum() to the mask when using RIFE, it gives a pretty good result.

Here's a sample frame with Blend / Rife / 2x Rife



Can do further tweaking on mask thresholds when using RIFE.

There are however some corner cases where RIFE gives worse results, such as quick spotlights. You might get 1 worse frame for 20 better frames.

[StainlessS]

MX,
Not totally sure, but think it was you who posted a K-Pop "Girl's Day - Female President" clip,
I loved that and have went out of my way to find more of GD. Also I found I liked "Itzy - Wannabee",
plus more of them too. but my favourite is probably "BlackPink - Kill this Love" [and their entire collection],
I also liked the "Making Of, Kill This Love" where they get showered with boulders [not the Boulder on this site]
during the explosion. Great stuff, and thanks for putting me on the this K-Pop thing, love it.

BlackPink - Kill This Love:- https://www.youtube.com/watch?v=2S24...=RD2S24-y0Ij3Y

EDIT: Yep I thought so, your triple image above this post is from "Female President":- https://www.youtube.com/watch?v=xF3MC8PWgJE

[MysteryX]

StainlessS, that's what the Natural Grounding Player is for, attuning to the energy of those videos. Although that software now has broken download feature and it will require a lot more work before the next version is ready. I recently published the whole database of videos here. Long list, and great stuff in there. Try this one and that one.

[poisondeathray]

Yes, RIFE can sometimes produce worse results on seemingly "simple" content that mvtools2 breezes through without issues

I've been doing quite a bit of testing and there are frames where 2.4 does better job than 3.1 or 3.8. I'm finding 2.3 seems the best model overall on varied content (but it's not included in vapoursynth version - But YMMV as usual . Scene changes are not handled well either - probably the same underlying issue with those flashing lights

Also , often you can get a "clean solve" on some frames by pre manipulations such as turnleft , or flipping (then reverting the change post rife) ; so I asked zorr about zopti maybe figuring combinations out on a per frame basis

It would be pure awesomeness if an automatic tiered choice algorithm could choose what works based on least artifacts, then fallback on some of the masking, or full blending if none work . MVTools2 is much faster still, and if it works - great . That's my preference. But if that doesn't work, replace that frame with RIFE 2.4 , and if that doesn't work replace with xyz etc...

[MysteryX]

Originally Posted by poisondeathray

It would be pure awesomeness if an automatic tiered choice algorithm could choose what works based on least artifacts, then fallback on some of the masking, or full blending if none work . MVTools2 is much faster still, and if it works - great . That's my preference. But if that doesn't work, replace that frame with RIFE 2.4 , and if that doesn't work replace with xyz etc...

That's pretty much what I'm doing; except that RIFE doesn't provide any info as to what artefacts it produces. Full-frame blending or skipping is done based on the raw masks from MvTools. Which handles scene changes correctly.

If there remains specific situations causing trouble, the only way would be to have a specific plugin to scan for that problem, just like StripeMask is doing.

BTW can someone explain this. I run twice with output="raw" debug=true in VapourSynth, and although the raw mask visually looks the same, the raw value varies randomly. Why does Vpy MvTools have this "random" factor?

I confirm that model=1 gives better results. The random junk that sometimes appear when running multiple instances does make it harder to test. Also, VirtualDub process doesn't terminate correctly. It does have a few bugs that need to be rounded up.

[StainlessS]

Thanks for that MX, I'm aware of some of them.
Love this [Japanese] one from Kill Bill II [closing credits], "Urami Bushi" by Meiko Kaji:- https://www.youtube.com/watch?v=yT9zJ2V9lfw

EDIT: Didn't realize it but Meiko Kaji has serveral more tracks from Kill Bill I.

[MysteryX]

Beta 11

Added "rife" and "rifeanime" presets
Preset "rife" sets rife=2 and maskThr=80. If doubling framerate, rife is set to 1-pass.
Rife uses model=1 (v2.4)
Changed maskOcc from 105 to 125, and blending with .7 opacity instead of .4
Avisynth version now has FullRange parameter

Using diff mode with RIFE is counter-productive, so preset "rife" uses preset "slow".

Remains a weird randomness to the masks. Run the same clip twice (with preset="normal") and you'll get slightly different masks.

I created a thread for RIFE problems here. Reported the MvTools randomness here.

This now gives seriously good results!! Please test it and post your reports.

Pay particular attention to the occlusion mask. VapourSynth occlusion mask is a lot narrower than in Avisynth where it marks large areas. If you find that occlusions aren't being detected enough, yet show on the raw mask, let me know and we can tweak either maskOcc or the merge opacity. Also do a bit of testing about StripeMask as to whether it includes too much.

So please test these 3 areas and let me know whether it feels weak, just right or too strong.
- Occlusion mask
- Stripe mask
- Overall mask

[poisondeathray]

Thanks for the updates

Originally Posted by MysteryX

Rife uses model=1 (v2.4)

That was for v1, which is loaded with a .dll
model: Model to use.

0 = rife-v3.1
1 = rife-v2.4
2 = rife-anime



v1.1 uses model_ver
3.1
3.5
3.8

You can check __init__.py

v1.1 is accessible when you use pip install method


To access all the models for the vapoursynth version, I've been both loading the old R1 .dll and importing the R1.1 py. I've been using stackvertical/horizontal to check versions, also other non vapoursynth versions to check other model versions

core.std.LoadPlugin(r'PATH\RIFE.dll') #RIFE-r1
from vsrife import RIFE as RF3 #RIFE r1.1

r24 = core.rife.RIFE(clip , model=1)
ranime = core.rife.RIFE(clip , model=2)
r31 = RF3(clip , model_ver=3.1) #3.1,3.5,3.8
r35 = RF3(clip , model_ver=3.5)
r38 = RF3(clip , model_ver=3.8)

Remains a weird randomness to the masks. Run the same clip twice (with preset="normal") and you'll get slightly different masks.

Did you check source filter, maybe try threads=1

[poisondeathray]

I can't get vapoursynth version to run, never seen this error before

File "FrameRateConverter.py", line 170, in FrameRateConverter
Blank = core.std.BlankClip(C.resize.Point(format=vs.GRAY8))
File "src\cython\vapoursynth.pyx", line 2067, in vapoursynth.Function.__call__
vapoursynth.Error: BlankClip: nodes foreign to this core passed as input, improper api usage detected

The py says Version: 2.0 (2021-09-18) beta 9 , not beta 11 - could it be wrong bundled version ?

EDIT - NM , some "weirdness" going on , but you need to specify output mode and blkSize, but sometimes that error message comes back... weird . You have to close vsedit and restart and message goes away. It's really flaky right now, maybe it's related to that weird mask randomness. Similar thing happens in vdub2 - 1st time works, but if you push f2 (reopen video file) that error message pops up

Selur had the same issue a few pages back
https://forum.doom9.org/showpost.php...&postcount=402

Might be related
https://forum.doom9.net/showthread.p...28#post1801028
https://forum.doom9.net/showthread.p...33#post1801033

If I edit the FrameRateConverter.py to

from vapoursynth import core

instead of

core = vs.get_core()

the issue goes away

[MysteryX]

Ah yes there was that. Probably related to the way I'm playing with Core instance in StripeMask code. I remember doing a hack I wasn't sure about.

vsrepo gives me v1.0 I believe? Where do I find that v1.1?

[ChaosKing]

I belive he means https://github.com/HolyWu/vs-rife
These are 2 different plugins

vsrepo currently only has https://github.com/HomeOfVapourSynth...FE-ncnn-Vulkan
Not sure if vs-rife is worth adding if you still have have many post install steps.