Frame interpolation using NNEDI3?

Soulvomit · 27th December 2020, 15:17

This idea occurred to me but I'm having a hard time executing it. When used as a deinterlacer, NNEDI3 works spatially to create new fields, but how can it be used temporally as a frame interpolator? I know it isn't intended for this purpose but I'm really curious of the results. So what would that script look like?

Sharc · 28th December 2020, 14:51

Spatial and temporal interpolation are based on different principles.
I don't think you can 'abuse' nnedi3 for temporal interpolation just by means of a script.
For temporal interpolation see the 'Frame Rate Conversion' section in avisynth's External Filters.

johnmeyer · 28th December 2020, 16:08

Quote:

Originally Posted by Sharc

Spatial and temporal interpolation are based on different principles.

Absolutely true, but don't the "better" deinterlacers, like QTGMC, do both? It uses MVTools2 to estimate a field at an intermediate point in time, shifted up or down by one scan line.

Frank62 · 28th December 2020, 16:58

I thought so, too. But I fear there will be no way to produce something "in between" two frames. You could simply produce pseudo-interlaced frames by taking a clip and the same shifted one frame, maybe like this:

source=xxsource("file")
shift=trim(source,1,0)
i=interleave(source,shift)
i=weave(i)

But if you deinterlace this, the interpolation will always be optimized to the one field you specify. If temporal methods are used like with QTGMC, then full motion-compensation is used to produce a second clip for interpolation, but still for the exact same point in time. If no temporal methods are used, interpolation is fully spatial, so the second field won't be used at all. The idea, though, is interesting somehow.

Soulvomit · 29th December 2020, 09:46

Yeah, I know there are better, proper ways to interpolate frames. This is just strictly an experiment of using NNEDI3's way of creating new fields to create new frames instead. Let's assume the source is a 640x480 progressive RGB video. You would have to first separate it into individual fields or rows of pixels. For the first field/row you would crop vertically (crop(0,0,0,-479), then stack those rows vertically with one another, then double the height of that stack with point resize, then deinterlace that stack with NNEDI3(field=1) to get the new fields/rows, then separate/unstack those new fields/rows, before finally stacking them again with the other new ones after doing this 479 other times and interleaving that with the source. It's the stacking and unstacking I'm stumped on.

Frank62 · 29th December 2020, 13:31

I didn't want to show that there are better ways to interpolate. I only used QTGMC as an example, I know better. I don't know exactly how nnedi3 works, but I wanted to show that:

IF it does interpolation spatially then there can be no temporal benefit at all.
IF nnedi tries to interpolate temporally from neighbour frames then this must be done similar to QTGMC:
1. FULL motion compensation of the neighbour frames
2. Use these compensated pixels from neighbour frames in some way
The problem for your task is that pixels that are temporally BETWEEN frames will never be generated.
Can't say it better in English.

Soulvomit · 29th December 2020, 14:17

I'm with you. I just used the word temporally very loosely just to refer to using NNEDI3 horizontally instead of vertically using adjacent frames for the creation of new ones. Technically this is still a spatial affair as that's the only way NNEDI3 works on its own and nothing else will be used, I'm very aware of that, but even so I'm curious to see the results of this experiment.

27th December 2020, 15:17	#1 \| Link
Soulvomit Registered User Join Date: May 2012 Posts: 28	Frame interpolation using NNEDI3? This idea occurred to me but I'm having a hard time executing it. When used as a deinterlacer, NNEDI3 works spatially to create new fields, but how can it be used temporally as a frame interpolator? I know it isn't intended for this purpose but I'm really curious of the results. So what would that script look like?

28th December 2020, 14:51	#2 \| Link
Sharc Registered User Join Date: May 2006 Posts: 4,015	Spatial and temporal interpolation are based on different principles. I don't think you can 'abuse' nnedi3 for temporal interpolation just by means of a script. For temporal interpolation see the 'Frame Rate Conversion' section in avisynth's External Filters. Last edited by Sharc; 28th December 2020 at 14:59.

29th December 2020, 09:46	#5 \| Link
Soulvomit Registered User Join Date: May 2012 Posts: 28	Yeah, I know there are better, proper ways to interpolate frames. This is just strictly an experiment of using NNEDI3's way of creating new fields to create new frames instead. Let's assume the source is a 640x480 progressive RGB video. You would have to first separate it into individual fields or rows of pixels. For the first field/row you would crop vertically (crop(0,0,0,-479), then stack those rows vertically with one another, then double the height of that stack with point resize, then deinterlace that stack with NNEDI3(field=1) to get the new fields/rows, then separate/unstack those new fields/rows, before finally stacking them again with the other new ones after doing this 479 other times and interleaving that with the source. It's the stacking and unstacking I'm stumped on. Last edited by Soulvomit; 29th December 2020 at 10:06.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

28th December 2020, 16:58	#4 \| Link
Frank62 Registered User Join Date: Mar 2017 Location: Germany Posts: 251	I thought so, too. But I fear there will be no way to produce something "in between" two frames. You could simply produce pseudo-interlaced frames by taking a clip and the same shifted one frame, maybe like this: source=xxsource("file") shift=trim(source,1,0) i=interleave(source,shift) i=weave(i) But if you deinterlace this, the interpolation will always be optimized to the one field you specify. If temporal methods are used like with QTGMC, then full motion-compensation is used to produce a second clip for interpolation, but still for the exact same point in time. If no temporal methods are used, interpolation is fully spatial, so the second field won't be used at all. The idea, though, is interesting somehow.

29th December 2020, 13:31	#6 \| Link
Frank62 Registered User Join Date: Mar 2017 Location: Germany Posts: 251	I didn't want to show that there are better ways to interpolate. I only used QTGMC as an example, I know better. I don't know exactly how nnedi3 works, but I wanted to show that: IF it does interpolation spatially then there can be no temporal benefit at all. IF nnedi tries to interpolate temporally from neighbour frames then this must be done similar to QTGMC: 1. FULL motion compensation of the neighbour frames 2. Use these compensated pixels from neighbour frames in some way The problem for your task is that pixels that are temporally BETWEEN frames will never be generated. Can't say it better in English.

29th December 2020, 14:17	#7 \| Link
Soulvomit Registered User Join Date: May 2012 Posts: 28	I'm with you. I just used the word temporally very loosely just to refer to using NNEDI3 horizontally instead of vertically using adjacent frames for the creation of new ones. Technically this is still a spatial affair as that's the only way NNEDI3 works on its own and nothing else will be used, I'm very aware of that, but even so I'm curious to see the results of this experiment.