Interpolation functions (FrameSurgeon, Morpheus, Morph)

[abolibibelot]

I gave yet another shot at improving this 4 years old video of my grandma (already mentioned here and here) for what is hoped to be the final version of an insanely long running project. For this I had to make intensive use of Avisynth frame interpolation functions to fix an almost constant jerkiness / blurriness defect introduced in the raw footage by a defective image stabilizer : mainly FrameSurgeon, seconded by Morpheus and Morph for some tricky frames which were not satisfyingly processed by FrameSurgeon.

– As a general comment on the overall aspect of the interpolated frames, those produced by Morpheus tend to be slightly less precise while producing less egregious artifacts, compared with those produced by FrameSurgeon, which may or may not be preferred depending on each particular frame. When the adjacent frames are not too dissimilar and when there's little motion involved, both produce excellent and virtually indistinguishable result. Morph produces a markedly different output in that it tends to blend adjacent frames rather than attempting to interpolate the missing one ; in somes instances it may look cleaner on a still frame but most of the time it's more disturbing when seen in motion than an interpolated frame with moderate motion artifacting (especially when Morph is used to repair several frames in a row, or several individual frames located in close succession, especially if a large object is moving at the forefront, for instance if a person is walking right in front of the camera, and several frames are processed by Morph, it gives the impression that the person is teleporting rather than materially moving... or if the frame itself is moving rapidly it can look like a crossfade between two distinct clips) – although it may still be preferred when the interpolation artifacts are very conspicuous (in some cases as a last resort I pasted together several versions of a particularly tricky frame with GIMP and inserted the PNG output back into the script with RemapFrames – which is definitely overkill as noone but me will would have noticed and complained, or even cared). It also has a stability drawback : it can choke when more than a few dozen Morph commands are called within a script, and can cause a crash of some kind (Avisynth crash, not a system wide crash). I've extensively tested quite a few others interpolation functions before sticking to those three, which are very efficient in their own right and complimentary.

– According to "StainleSS", Morpheus was conceived to handle thousands of interpolation commands, yet in my experience it isn't as slick as FrameSurgeon in that regard. In particular, it generates a LOT of temporary files, and these ~_MORPHEUS_X_YYYYMMDD_HHMMSS_XXX.DB files are quite huge, 1344KB for most of them (it may not sound like much but when there are thousands it begins to add up to an awful lot), while a new bunch seems to be generated each time the script is reloaded in AVSPMod or any other utility, currently 132 new files for a total of 173MB (with 125-130 Morpheus commands in the script), to the point that it repeatedly filled up the working partition (a 100GB partition on a SSD), and, not knowing if I could delete them altogether, I resorted to apply NTFS compression on them, which reduced their size to 8KB each (those files are actually almost empty). When I noticed that issue in September I noted the total number of files and corresponding sizes, for each day :

2019/09/22 : 431 files, total size = 520MB, 3.23MB with NTFS compression
2019/09/07 : 2140 files, total size = 2.73GB, 16.6MB with NTFS compression
2019/09/10 : 59 files, total size = 76.3MB, 468KB with NTFS compression
2019/09/16 : 470 files, total size = 614MB, 3.66MB with NTFS compression
2019/09/17 : 1298 files, total size = 1.66GB, 10.1MB with NTFS compression
2019/09/18 : 813 files, total size = 1.04GB, 6.35MB with NTFS compression
2019/09/22 : 4571 files, total size = 5.85GB, 35.7MB with NTFS compression
2019/09/23 : 4873 files, total size = 6.24GB, 38.0MB with NTFS compression
2019/09/24 : 3844 files, total size = 4.92GB, 30.0MB with NTFS compression
2019/09/25 : 4426 files, total size = 5.67GB, 34.5MB with NTFS compression

And now I put all those files in a subfolder which amounts to a total of 70874 files for a whopping 90.4GB (574MB with NTFS compression).
There should be a way to reuse them the next time around when the same commands are used (or if a single one is added for instance), if at all possible with no significant programming or performance issue, or at least Morpheus should apply the NTFS compression automatically on those files, and put them in a subfolder right away, so as to avoid clogging the main directory which contains the scripts and other related files which need to be easily accessible.
By contrast, the temporary files generated by FrameSurgeon / ClipClop are much smaller, 35KB at most, and only a few are created during each session (besides they only seem to contain a copy of the list of commands).

– When running the script in VirtualDub2 x64 through Avisynth+ 0.1 r2772 x64, I noticed that some frames processed by Morph (not all of them) {=> actually all of them : see below} were wonky – it looks like the n-1 frame (for instance frame 12345 with the command Morph(12345,12347) which is supposed to re-create frame 12346) but dimmer and with weird patterns in some of the lighter areas. Is there any obvious explanation ? According to member "StainleSS" in this other thread, "Jenyok functions sometimes have 'undocumented features', and tend to be a bit big and cumbersome" ; could it be related with this issue ?
Illustrations :


The first screenshot is taken from VirtualDub2 x86, the frame is correctly processed (in this case two frames are processed in a row with the command Morph(23302,23305)) ; the second screenshot is taken from VirtualDub2 x64 (using the PullInterpFrames analysis function which allows to review only the interpolated frames),

– A different issue, not directly related with interpolation functions, although it did cause me trouble when applying them : using LWLibavVideoSource to load M2TS video files, I found out that it would display a few frames wrong, typically as a duplicate of the next one. With FFVideoSource the same frame appears normal. And if one of those frames (they are few and far between, with no obvious pattern) is interpolated with FrameSurgeon, it can produce that kind of wonky output :


Here the first screenshot shows frame #239 after a FrameSurgeon interpolation of 3 consecutive frames with the command I3 237 ; the second case is even weirder as it shows frame #5410 after the interpolation of the next one, #5411, with the command I1 5411. If I remember correctly, disabling the FrameSurgeon command, reloading the script, then enabling it again and reloading again would correct the display.
So, is LWLibavVideoSource known to be unreliable ? If so, is it a general problem, or does it affect some file types in particular ? Is there a list of recommended source plugins for each video format, or a list of the known behaviour of each one with its advantages and drawbacks depending on the situation ? (If not, there should !)

Thanks for any insight (if hindsight is 20/20, how much is insight ?... and oversight ?...).


EDIT : Actually, I verified again, all the Morph commands produce an abnormal output through Avisynth+ 64 bits.


{*} As a side note, I don't understand why those PNG screenshots have to be recompressed to JPG, as their size is only about 800KB, which is not that much by nowadays standards.

[StainlessS]

Well firstly,

thousands of temp files

Morpheus() v0.02, Just like you dreamed it to be

Interpolate frames using a Command list of frames in String or in file.

Req RT_Stats v1.43+, Avs+ or GScript, Grunt, MvTools, RemoveGrain, CallCmd (optional, auto deletes DBase).

the second case is even weirder as it shows frame #5410 after the interpolation of the next one, #5411, with the command I1 5411.

Not sure, think there was a problem with older MvTools which produced a wrong frame from old cache, or something like that. Update mvtools.

So, is LWLibavVideoSource known to be unreliable ?

Sometimes.

I nearly always use an ffmpeg bat script to convert to AVI, avoids all manner of problems with source filters.

REM We DO NOT LIKE SPACES IN FILE NAMES (REM == REMark ie comment)

setlocal

REM Where to Find ffmpeg
set FFMPEG="C:\BIN\ffmpeg.exe"

REM Where to get input file, No terminating Backslash, "." = current directory (ie same as dir .bat file)
set INDIR="."

REM Where to place output file, No terminating Backslash. "." would be same as .bat file, ".\\OUT" = OUT folder in same directory as bat file.
set OUTDIR="D:"

REM Below, can add extensionas as eg *.WMV (SPACE separated)
FOR %%A IN (*.mp4 *.vob *.mpg *.TS *.MOV) DO (

REM ****** Un-REM ONLY one of below lines *******.
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec copy    -acodec copy      "%OUTDIR%\%%~nxA.MKV"
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec utvideo -acodec copy      "%OUTDIR%\%%~nxA.MKV"
    %FFMPEG% -i "%INDIR%\%%A" -vcodec utvideo -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"
REM *********************************************.

)

REM ... Above UN-REM'ed lines :
REM      (1) Remux, copy both video and audio   (output MKV).
REM      (2) UtVideo lossless video, copy audio (output MKV).
REM      (3) UtVideo lossless video, PCM audio  (output AVI).

Pause

Weirdness,
well MC stuff sometimes does that, especially if not frame accurate source. [EDIT: yep, some of your posted images look more weird than expected, see remainder of this post]

Problem in Mvtools[see from link post to end of thread]:- https://forum.doom9.org/showthread.p...16#post1889616
I dont think I would be in a hurry to do any MvTools stuff until above linked prob fixed, it has been broken for long time, perhaps since the 1st release.


EDIT: I suggest that if using Morpheus with FrameSurgeon, to actually use the Morpheus produced clip, not the morpheus script [as FXd clip source], would be a lot faster,
and I guess that is the reason that you had lots of old temp DBase files, because you were doing single frame edits/vdub refresh, with a new temp DBse file created on every refresh.
EDIT: Using Morpheus produced clip, would also avoid many Morpheus restarts in middle of clip, is likely more reliable if start from the beginning of clip, I think Morpheus was written to be able
to be started at any point in clip, but it may not be bullet proof. [EDIT: Especially if restarting at bad frames, which is probably what you are doing when using Sawbones/FrameSurgeoun to
open Morpheus script].

EDIT: Update mvtools, and
avs+ v3.4.0:- https://github.com/AviSynth/AviSynthPlus/releases

[abolibibelot]

Thanks for the quick feedback.
The installed version of mvtools is 2.7.37.0 from 2018/12/06. There are so many moving parts in this thing and so many unexpected SNAFUs that I'm now very wary of updating anything until it's done and over with, fearing that changing anything could mess with the outcome of the meticulously defined interpolation commands for each one of the painstakingly identified defects, after staring at some of the ugliest frames for so long that they're etched into my sorry soul and I could draw the patterns of the interpolation artifacts from memory... é_è (At this point it would be almost easier to recreate the whole footage as a CGI cartoon.)

Req RT_Stats v1.43+, Avs+ or GScript, Grunt, MvTools, RemoveGrain, CallCmd (optional, auto deletes DBase).

Indeed, I hadn't noticed this option. So it's totally safe to delete them ? What are these files for anyway, and how come FrameSurgeon doesn't require them ? (You're the author of both if I'm not mistaken – the Wizard of Interpolation !)

Not sure, think there was a problem with older MvTools which produced a wrong frame from old cache, or something like that. Update mvtools.

Don't call me Not Sure ! è_é As I mentioned, there's no such issue when loading the source file with FFVideoSource, so it seems to be specifically caused by LWLibavVideoSource.

I nearly always use an ffmpeg bat script to convert to AVI, avoids all manner of problems with source filters.

I see a choice in this script between AVI and MKV, so is MKV generally considered a more reliable container than MP4 or M2TS with regard to Avisynth interaction ?

Weirdness,
well MC stuff sometimes does that, especially if not frame accurate source.

What do you mean by "MC stuff" ? (It might be obvious but I'm quite tired right now, and aching in places where I used to play...) Is LWLibavVideoSource supposed to be frame accurate ? How is it different from FFVideoSource ? (I thought that ffmpeg and libavcodec were closely related projects, so I figured that, based on their names, FFVideoSource and LWLibavVideoSource were pretty much the same and mostly interchangeable. Apparently not.)

I suggest that if using Morpheus with FrameSurgeon, to actually use the Morpheus produced clip, not the morpheus script [as FXd clip source], would be a lot faster,
and I guess that is the reason that you had lots of old temp DBase files, because you were doing single frame edits/vdub refresh, with a new temp DBse file created on every refresh.

Well, do you mean at the encoding stage ? Could it be one explanation of the slow encoding speed I experienced today, as reported here ?
Before that, it's tricky as I had to review each frame and, for those which didn't work right away with FrameSurgeon (which I used first, before Morpheus even existed it would seem as I began this interpolation work almost two years ago in January 2018 – I hope I can finally finish this before everyone involved has died in battle or in vain), try another function or another setting (for instance interpolating 2 frames instead of 3, or sometimes 4 instead of 2 works better if the adjacent frames are "cleaner" or better "aligned", and so on) ; so the potential performance improvement wouldn't be worth the loss of the convenience of real time adjustment of all possible variables.
Also :
– There are currently 132 files created at each refresh (using either AVSPMod or VirtualDub). Here is the last batch, generated today at 19:50 (173MB compressed to 45KB) :
Attachment 17106
– I did the bulk of the interpolation work (using Sawbones / FrameSurgeon plus Morph for a few frames) when pre-processing intermediate files (some of which required thousands of interpolations), which I then imported into the editor, and lately I've been adding some extra commands to fix frames which I had missed or overlooked the first time around.

Using Morpheus produced clip, would also avoid many Morpheus restarts in middle of clip, is likely more reliable if start from the beginning of clip, I think Morpheus was written to be able to be started at any point in clip, but it may not be bullet proof. Especially if restarting at bad frames, which is probably what you are doing when using Sawbones/FrameSurgeoun to
open Morpheus script].

I'm not sure what you mean here exactly, as it seems to refer to technical details of each function's operation. At any rate, it does work remarkably well considering how complex it all is, and despite those little hiccups the result is mostly excellent.

I wanted to upload the script here but I get an error message saying : “You have already attached this file in thread : Avisynth, frame interpolation, ffmpeg x264, performance concerns” – so you'll find it in that thread, along with the FrameSurgeon command file.

[StainlessS]

Firstly, update to current version mvtools, :- https://github.com/pinterf/mvtools/releases
and avs+ 3.4.0:- https://github.com/AviSynth/AviSynthPlus/releases
Nobody likes debugging probs that have already been fixed.

Secondly, this lot

LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins\Prune26.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins\RT_Stats26.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins\RemapFrames32.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins\ClipClop_x86.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins\GRunT.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins\RgTools.dll")

Import("C:\Program Files (x86)\AviSynth+\plugins\FrameSurgeon.avsi")
Import("C:\Program Files (x86)\AviSynth+\plugins\Morpheus3.avs")                   # <<<<<<<<<<<<<<  Change extension to .avsi, and will auto inport from your plugs directory.
Import("C:\Program Files (x86)\AviSynth+\plugins\Morph.avsi")

All of above will autoload from your plugins (see above Morpheus3.avs comment), so above not necessary.

Originally Posted by abolibibelot

I see a choice in this script between AVI and MKV, so is MKV generally considered a more reliable container than MP4 or M2TS with regard to Avisynth interaction ?

Sorry for any confusion, that was just a demo bat, how to use in different way, eg for demux.
I would also convert MKV files to AVI (you never know what is inside MKV unless you look with eg MediaInfo).
The only container that I consider reliable ie frame accurate, is AVI [assuming that it looks ok on load].

Originally Posted by abolibibelot

What do you mean by "MC stuff" ? (It might be obvious but I'm quite tired right now, and aching in places where I used to play...)
Is LWLibavVideoSource supposed to be frame accurate ? How is it different from FFVideoSource ? (I thought that ffmpeg and libavcodec
were closely related projects, so I figured that, based on their names, FFVideoSource and LWLibavVideoSource were pretty much the same
and mostly interchangeable. Apparently not.)

MC=Motion Compensated.
FFVideoSource and LWLibavVideoSource, may or may not be frame accurate for a specific video file, other than that who knows.
One of them (FFVideoSource or LWLibavVideoSource) gives a list in its thread for containers considered frame accurate, and some that might be frame accurate
but I dont 100% trust them for any file. [you sometimes have threads with many many posts over several weeks, on how to best open a specific single file,
life is really too short for such nonsense].

Originally Posted by abolibibelot

Well, do you mean at the encoding stage ?

Your script, 64 lines similar to

RemapFrames(last, sourceClip=ImageSource("G:\20131224 M2TS\20151224 017162 collage 3.png", pixel_type="RGB32"), mappings="17162 0")

Are not a big problem, but I personally (in a big-ish complex-ish project) would do it in stages, ie save result of the remapFrames as a single YV12 file,
yes it might take a while to write the new file, but you dont have to supervise it.

I found the later script [fragemnt of it below] a little startling.

FrameSurgeon(Cmd="G:\20131224 M2TS\20151224 pour encodage [20190730] FrameSurgeon.txt", Show=false)

#Morpheus("*17167")
#FrameSurgeon("I1 17167") OK
#Morpheus("*17169")
Morpheus("*17193")
Morpheus("*17277")
Morpheus("*17280")
Morpheus("*17292")
#FrameSurgeon("I1 17691")
Morpheus("*17691")
Morpheus("*18493")
#Morpheus("*18930")
#Morpheus("*19413 19415")
#FrameSurgeon("I1 19418") => capture d'écran et collage GIMP
#FrameSurgeon("I2 19434") => capture d'écran et collage GIMP
Morpheus("*19503 19504")
#Morph(19509,19512)
#FrameSurgeon("I1 19806") => capture d'écran et collage GIMP
#Morpheus("*19806")
Morpheus("*19861")
#Morph(19878,19880)
#Morpheus("*22954")
#Morpheus("*22962")
Morpheus("*23059 23061")
Morpheus("*23134")
#Morpheus("*23139")
#Morpheus("*23143")
Morph(23142,23144)
Morpheus("*23168")
#Morpheus("*23170 23173")
Morpheus("*23170 23171")
Morpheus("*23208")
Morpheus("*23213 23219")
#Morpheus("*23259")
#Morpheus("*23273 23281")
Morpheus("*23288")
#Morpheus("*23290 23291")
#Morpheus("*23294")
Morph(23293,23295)
#Morpheus("*23303 23304")
Morph(23302,23305)
#Morpheus("*23309 23311")
Morph(23308,23312)
#Morpheus("*23323")
#Morph(23322,23324)
#Morpheus("*23325 23326")
#Morph(23324,23327)
#Morpheus("*23329")

Morpheus, was written to handle thousands of interpolated frames, but in a single call. It was never anticipated that anybody would
call the function replacing a single frame, hundreds of times, this is the reason that you have gazillions of temp DBase files [When CallCmd is not available to auto delete them].
Calling Morpheus many times with single frames is a recipe for disaster, especially if non frameaccurate source filter.
[I feel kinda pleased with myself that it dont crash every single time you press the refresh key, 100+ clips reopened every time with 100+ Dbases also created every time, WOW!!!]

Again, I think that you should process in stages, eg

Produce stage_1, with all of your Remapframes stuff. [EDIT: I guess that its no big deal to leave as it is for the RemapFrames thing, esp if 120GB source.]
Stage_2, FrameSurgeon result.
Stage 3, Morpheus USING SINGLE COMMAND FILE of frames to interpolate.
Stage 4, any single frame Morph() whatsits.

You can use as input, the previous stage, and a stackHorizontal result of next stage [editing command file as you add more, or viewing already added frames].

EDIT: the only reason that FrameSurgeon/Morpheous/Morph frames are different are the mvtools arguments used by each [assuming frame accurate source].

[abolibibelot]

All of above will autoload from your plugins (see above Morpheus3.avs comment), so above not necessary.

For some reason it works indeed with AVSPMod and VirtualDub2, but I had to add those lines for ffmpeg to properly load the script.
It may be because I used the AvisynthInstaller script at some point, which allows to switch between different versions, but perhaps also messed with the registered location of the main plugins directory. Or there's some other unfathomable weirdness involved. When I used PullInterpFrames in VirtualDub x64 I also had to add the corresponding lines to load the 64 bits versions of the required plugins (which are located in the "plugins64" Avisynth+ subfolder).
Anyway, while I understand that it's good practice to get things as streamlined as possible, it shouldn't hurt anything to have those lines in the script, right ?

Morpheus, was written to handle thousands of interpolated frames, but in a single call. It was never anticipated that anybody would
call the function replacing a single frame, hundreds of times, this is the reason that you have gazillions of temp DBase files [When CallCmd is not available to auto delete them].
Calling Morpheus many times with single frames is a recipe for disaster, especially if non frameaccurate source filter.
[I feel kinda pleased with myself that it dont crash every single time you press the refresh key, 100+ clips reopened every time with 100+ Dbases also created every time, WOW!!!]

Again, I think that you should process in stages, eg

Produce stage_1, with all of your Remapframes stuff. [EDIT: I guess that its no big deal to leave as it is for the RemapFrames thing, esp if 120GB source.]
Stage_2, FrameSurgeon result.
Stage 3, Morpheus USING SINGLE COMMAND FILE of frames to interpolate.
Stage 4, any single frame Morph() whatsits.

Alright then... how stupid of me... O_o At first I did create a seperate text file but then I figured that it would be more convenient to have all those extra commands (Morph + Morpheus) in the script itself ; besides at first there were only a handful of them, it grew later on when I thoroughly verified the result and noticed that Morpheus performed better than FrameSurgeon on some tricky frames (for others it's the other way around). Perhaps this should be noted somewhere in the Morpheus file (in capital letters too). So I'll definitely take note of "stage 3" and adapt the script right away. As for separating each type of processing and rendering an intermediate lossless file for each one, would there be a definite advantage, considering the hassle it entails ?

[StainlessS]

would there be a definite advantage, considering the hassle it entails ?

Well probably stability, and less likelyhood of weird frames produced [ like you posted ] which I'm guessin' may disappear
unless totally down to frame inaccurate source.

Also, the Morph() calls [as I think you found out] will crap out and cause probs when you get above a certain number,
In a project I did about 2009, with ~2000 single frame edits, it would crash when over about 70 calls similar to Morph(),
I did that project in about 12 stages, there was no alternative.

EDIT: https://forum.doom9.org/showthread.php?t=176878
Above, Atak_Snajpera SeekTester, maybe of use. [test frame accurate source on clip]