Nyquist-limited video transitions generation

[DTL]

It is a post to core avisynth developers (if any still exist) and plugins developers.

For ability of using avisynth in mature video-processing pipelines it is required for developers to think about requirement of so-called Nyquist-limited data (video data, that coded video images).

I think old avisynth developers was PC-based and think in pixel-based domain of (old) computer pixel-based images. So core data transformations in avisynth was designed not for video-domain but (separate)pixel-domain. And many core avisynth filters creates non-Nyquist-limited output (badly conditioned in terms of Nyquist limits and Gibbs phenomenon - https://en.wikipedia.org/wiki/Gibbs_phenomenon), illegal for video data processing.

Sort of definition - pixel-domain images are coded by separate pixels, video-domain images are coded strictly by an array of (interconnected) samples. In strict math every sample of video image data has (more or less) influence on every point of decoded (optical) image. Close to practise many neibour samples has influence on points of decoded image close to these samples in space.

There exist a ways of workarounds to get more quality data, but it takes lines of scripting and processing power (and memory) for works with full arrays data instead of edges data only.

To re-write most of avisynth to video-domain is a hard enough task but a few filters can be fixed more easily.

Most commonly using is LetterBox filter for example. It currently works as filling stated number of samples/pixels with defined color. But it do not pay attention on Nyquist-limiting requirement on a transition from filled area to source video area. So if we try to feed its output to correct high-quality (reference) video monitor we got ringing artifact in worst case and contouring (halo/overshooting) artifact in mid-quality general public tv devices with smaller kernel size resamplers. Also the apperaing of artifact depends on video content on new created transition between video data and letter-boxed color data.
From the definition of reference video control monitor - https://tech.ebu.ch/docs/tech/tech3320.pdf
1.5.19 Ringing and handling of under- and over-shoots
Ringing or overshoots should not be introduced by any processing in the monitor if the input signal is
suitably conditioned (i.e. is within Nyquist limits).
1.5.20 Treatment of illegal signals
Signals which contain significant frequency components outside the Nyquist limit (i.e. have not been
anti-alias filtered) should be displayed in such a way that any ringing or aliasing inherent in such
signals DOES become visible.


To fix it we require to pay more attention at forming new transition in video samples array data when creating such simple operation as 'letterbox'. One simple solution is apply low-pass filtering on new transition, with limited radius like 3 or 5 samples around its center. It is not great (reference) quality solution but already helps alot.

So in the best all core avisynth filters that can be source of new changes in frequency-domain in video data must be analysed and fixed as possible. May be in avisynth+. Mostly affected filters are geometric deformers: Crop, LetterBox, AddBorders.

Example of user-defined workaround for better conditioned LetterBox result:
inner=LetterBox(2,2,2,2)
mas=BlankClip(inner)
mas=LetterBox(mas,4,4,4,4,color=$ffffff)
mas=ColorYUV(mas, off_y=-20, gain_y=256)
outer=Blur(inner, 1.0)
Overlay(inner,outer,mask=mas)

It uses many more cpu and memory in compare with just more correctly designed core LetterBox filter itself. Ofcourse user may define separate library of user workarounds for many cases but it will slow-down system. If avisynth is designed for processing video data it is nice to have built-in core filters handling video data correctly.

Quick sample: Todays LetterBox works as

BlankClip(10,160,160,"YV12",25,color=color_gray90)
LetterBox(20,20,20,20,color=color_gray10)
SincResize(width*4,height*4,taps=20) // output to video monitor test

And better conditioned version may be close to

BlankClip(10,160,160,"YV12",25,color=color_gray90)
LetterBox(20,20,20,20,color=color_gray10)
GaussResize(width/2,height/2, p=30)
LanczosResize(width*2,height*2)
SincResize(width*4,height*4,taps=20) // output to video monitor test
It is just for show output quality of new added level transitions to video data, not as example of code realisation.