ONNX models to use with avs-mlrt

[DTL]

Some ideas: If AI/NN plugins typically accept only RGB in float - do the current models/kernels make some benefit in quality if being fed with linear RGB instead of transfer-encoded ? The float samples format allows us to feed linear data without significant quantization distortions. And plugins like Avsresize allows to do both integer to float conversion and YUV decoding and also transfer function decoding to linear RGB in a single call.

Each model/kernel may be trained with unknown dataset for the samples domain encoding - either some system transfer-encoded (601/709/HDR) or linear.

[poisondeathray]

Originally Posted by DTL

Some ideas: If AI/NN plugins typically accept only RGB in float - do the current models/kernels make some benefit in quality if being fed with linear RGB instead of transfer-encoded ?

Usually not, because 99.9% (probably 100%) of the models are trained on non linear data sets (and that includes 100's of other machine learning projects currently available but not as avs/vpy plugins) . Even commercial machine learning projects use gamma encoded sources

But you can train one on linearized data if you wanted to - I doubt it would provide any substantial benefit for current tasks

One area that would benefit from models trained on linearized datasets, is compositing. All high end compositing is done in 32bit linear environment. So inferencing/predictive models in that scenario could be useful

[simple_simon]

Are there any models that help with deblending bad pal-->ntsc conversions? The kind that QTGMC+srestore can't handle. Also any good ones for deblocking?

[SaurusX]

Originally Posted by simple_simon

Are there any models that help with deblending bad pal-->ntsc conversions? The kind that QTGMC+srestore can't handle. Also any good ones for deblocking?

There really has not been much advancement in that area. The best option to use is still animeivtc(mode=2), whether it's actually anime or live action.

[simple_simon]

It seems like a perfect use case for AI. Train it to recognize what a blend looks like (even outside of a regular pattern) and deblend just those frames.

[Selur]

Probably, but (nearly) all models in vs/avs-mlrt have no temporal component and only filter one image at a time (without taking the surrounding frames into account), I doubt an algorithm can decide on a single image on how the deblended version should look like.

[DTL]

Any good denoiser for moving pictures must be temporal. Several GBs GPU board can fit many frames for temporal processing.

Even 'simple' RIFE make processing based on 2 input frames. And we expect next RIFE will use more frames (to estimate not only 2-points speed but several points acceleration for better motion interpolation).

[Selur]

@DTL: So, your conclusion is that that most of the models/filters that can be used with vs-/avs-mlrt can't be good since they only work on a per frame bases?

[DTL]

It may be good if filter is not depends on data in the frame. Like ColorYUV or Levels. But if it depends on frame data - processing of several frames in a sequence may cause additional temporal non-stability.
But typical nature of NN/AI filters is very deep dependence on the data to process. So better to ask NN/AI engine developers to make multi-frame engines for better temporal stability.

[kedautinh12]

Originally Posted by DTL

It may be good if filter is not depends on data in the frame. Like ColorYUV or Levels. But if it depends on frame data - processing of several frames in a sequence may cause additional temporal non-stability.
But typical nature of NN/AI filters is very deep dependence on the data to process. So better to ask NN/AI engine developers to make multi-frame engines for better temporal stability.

True but the filters will take more vram and time to process all frames although it already take very much vram and time to process just some frames

[DTL]

HD/UHD frames are only MBs in size. Not GBs. So I think it is only still poor optimization of AI/NN for moving pictures processing at current hardware. As I see the main task for many current NN/AI developers is only make some working demo for 1 frame processing (for publication and level-up) but not working product as freeware. So they put close to zero efforts for optimizing for movies processing. As example of good marketing product I see Topaz VEAI - it works very good at typical workstation and very fast (in comparison wih may other freeware solutions).

[tormento]

Starting now to understand mlrt and my first script fails

LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
Import("D:\Eseguibili\Media\StaxRip\Apps\Plugins\AVS\DehaloAlpha\Dehalo_alpha.avsi")
Import("D:\Eseguibili\Media\StaxRip\Apps\Plugins\AVS\Dither\mt_xxpand_multi.avsi")
Import("D:\Eseguibili\Media\StaxRip\Apps\Plugins\AVS\FineDehalo\FineDehalo.avsi")
DGSource("M:\In\Monster ~R2J\1-01-B5.dgi",ct=4,cb=0,cl=4,cr=4)
DGTelecide(mode=1, pthresh=3.5)
DGDecimate()
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_W2x(backend="ort", model=1, noise=-1, scale=2)

gives me the error:

I don't know what 'height_' means.
(D:/Programmi/Media/AviSynth+/plugins64+/mlrt_ort-1.1.5-AsdG.avsi, line 203)
(D:/Programmi/Media/AviSynth+/plugins64+/mlrt_ort-1.1.5-AsdG.avsi, line 206)
(M:\In\Monster ~R2J\mlrt_waifu.avs, line 13)

Any help is welcome

[StainlessS]

Maybe take a look at mlrt_ort-1.1.5-AsdG.avsi, line 203. [find "height_"]
Show the full error line 203 from above.
height_ is not defined.

[tormento]

Originally Posted by StainlessS

Show the full error line 203 from above.
height_ is not defined.

I already looked at it. In that line there is a call to that variable and it's not probably been declared yet.

The question is if somebody else has that script working (I simply renamed it) or if it's a bug.

[Arx1meD]

When using mlrt_ncnn, some ONNX models give an error:

System Exception - Access Violation

What can give such error?

[DTL]

Try different drivers for GPU and lower settings if possible (like lower threads and all possible other adjustable resources). Or may be GPU RAM size too low.

[Arx1meD]

I have a NVIDIA RTX 2060 Super, RAM - 8 GB.
I noticed that only ESRGAN and Compact architecture models work. This is written in the notes to the chaiNNer program - readme-ov-file # onnx. Probably this could be the cause of the error.

[kedautinh12]

Originally Posted by Arx1meD

I have a NVIDIA RTX 2060 Super, RAM - 8 GB.
I noticed that only ESRGAN and Compact architecture models work. This is written in the notes to the chaiNNer program - readme-ov-file # onnx. Probably this could be the cause of the error.

8Gb isn't enough if you load fullHD and above with the strong .onnx file

[Arx1meD]

I process DVD and video with a frame size 640x480.

[FranceBB]

Originally Posted by Arx1meD

I process DVD and video with a frame size 640x480.

Does it also happen with fp16=true?
I also have 8GB GDDR5 in my NVIDIA Quadro P4000 and I had to add that to compensate 'cause otherwise it would perform calculations in 32bit float and run out of memory.