ONNX models to use with avs-mlrt

[Reel.Deel]

PyTorch models converted to ONNX to use with AviSynth+'s avs-mlrt plugin. Models were converted with chaiNNer (fp32 data type). Let me know of missing models to convert and I'll add them to the package.


Download: onnx-models-rd

Other repositories that include onnx models:

• Large collection of onnx models: https://github.com/styler00dollar/VS...ses/tag/models
  
• 2x_AnimeJaNai - https://github.com/the-database/mpv-...2x_animejanai/
  
• Phhofm models (some already onnx): https://github.com/Phhofm/models

Model database

• https://upscale.wiki/wiki/Model_Database
  
• https://openmodeldb.info/

Basic usage

FFVideoSource("YUV420Source.mkv")
ConvertBits(32)
ConvertToPlanarRGB()
mlrt_ncnn(network_path="1x_Dotzilla_Compact_80k_net_g.onnx", builtin=false)

You can omit the builtin parameter if the models are in the same location as the avs-mlrt plugin (see builtin and builtindir parameters in the docs).

If you have a low end GPU and trying to use the larger models, it helps to use the tilesize, overlap and fp16 options.

mlrt_ncnn(network_path=model, builtin=false, fp16=true, tilesize_w=width/4, tilesize_h=height/4, overlap_w=8, overlap_h=8)

Model conversion

Download chaiNNer and install the required dependencies. Load the corresponding nodes, connect them and then hit run by either pressing the play button at the top or F5.

[Dogway]

Woow, so great, ''thumbs up'', these are my go to for anime and manga. Still searching a good one for photos though.

Old Anime Upscale:
    1x_Dotzilla_Compact_80k_net_g.pth (dot crawl and rainbowing)
    2X_DigitalFilmV5_Lite.pth (sharpener for soft lines, no need to downscale before AI)
    2x_AnimeClassics_UltraLite_510K.pth (less reconstruction than above, similar to Waifu but for Anime. Color Shift)
    2x_LD-Anime_Skr_v1.0.pth (for ringing, rainbowing, aliasing)
Manga Upscale (B/W):
    4x_eula_digimanga_bw_v1_860k.pth
Magazines?
    4x-UniScaleV2_Moderate

[kedautinh12]

Originally Posted by Dogway

Woow, so great, ''thumbs up'', these are my go to for anime and manga. Still searching a good one for photos though.

Old Anime Upscale:
    1x_Dotzilla_Compact_80k_net_g.pth (dot crawl and rainbowing)
    2X_DigitalFilmV5_Lite.pth (sharpener for soft lines, no need to downscale before AI)
    2x_AnimeClassics_UltraLite_510K.pth (less reconstruction than above, similar to Waifu but for Anime. Color Shift)
    2x_LD-Anime_Skr_v1.0.pth (for ringing, rainbowing, aliasing)
Manga Upscale (B/W):
    4x_eula_digimanga_bw_v1_860k.pth
Magazines?
    4x-UniScaleV2_Moderate

Can it use for real world video??

[poisondeathray]

Originally Posted by kedautinh12

Can it use for real world video??

Usually the models are quite narrow in scope; typically they are only useful on the data set that they were trained on, or very similar material. But you can try it...

Originally Posted by Reel.Deel

Edit: if anyone has the "DeGif" model please share it. The link to it is not working right now (Public Transfer Limit Exceeded).

Which version ? There are older checkpoint versions and the most recent version I have is "2x_NMKD-DeGIF_210000_G.pth"

[Reel.Deel]

Originally Posted by poisondeathray

Which version ? There are older checkpoint versions and the most recent version I have is "2x_NMKD-DeGIF_210000_G.pth"

I guess the latest one. There is no version number on the upscale wiki.

[Reel.Deel]

Originally Posted by Dogway

Woow, so great, ''thumbs up'', these are my go to for anime and manga. Still searching a good one for photos though.

Old Anime Upscale:
    1x_Dotzilla_Compact_80k_net_g.pth (dot crawl and rainbowing)
    2X_DigitalFilmV5_Lite.pth (sharpener for soft lines, no need to downscale before AI)
    2x_AnimeClassics_UltraLite_510K.pth (less reconstruction than above, similar to Waifu but for Anime. Color Shift)
    2x_LD-Anime_Skr_v1.0.pth (for ringing, rainbowing, aliasing)
Manga Upscale (B/W):
    4x_eula_digimanga_bw_v1_860k.pth
Magazines?
    4x-UniScaleV2_Moderate

Thanks for the suggestions. I added them to the package. I'll provide a Dropbox link from now on since the size is already at 331mb. This way people can pick and choose only the models they want. I did not test these new models, let's hope they don't blow up

Edit: if anyone has the "DeGif" model please share it. The link to it is not working right now (Public Transfer Limit Exceeded).

[kedautinh12]

wow, so i think this will open new era about pytorch can port to avisynth

[poisondeathray]

I mention it because sometimes an earlier checkpoint produces better results. Here are all the checkpoints for DeGif that I have. Credits to NMKD (n00mkrad)
https://www.mediafire.com/file/2jzoq...+DeGIF.7z/file

Just curious :
1) how long does it take to convert ?
2) are results bit identical to the pytorch version?

[Reel.Deel]

Originally Posted by poisondeathray

I mention it because sometimes an earlier checkpoint produces better results. Here are all the checkpoints for DeGif that I have. Credits to NMKD (n00mkrad)
https://www.mediafire.com/file/2jzoq...+DeGIF.7z/file

Just curious :
1) how long does it take to convert ?
2) are results bit identical to the pytorch version?

Thanks for the models.

1) Depends on the size of the model. Small ones are quick (few seconds), the largest one I converted (64mb) took around a minute.

2) I have not compared them to the original version. It would be interesting to do so. But while researching how to convert between the two formats I only saw that mentioned once or twice. See here: https://www.reddit.com/r/MachineLear...model_to_onnx/ and here https://pytorch.org/docs/stable/onnx.html#limitations

I'm all ears if anyone has any suggestions.

This is the interface in ChaiNNer:

I'm sure there more options when using PyTorch directly but I have not experimented with that.

Edit: can someone test the 2x_LD-Anime_Skr_v1.0.pth model? It crashes on me.

[Dogway]

Originally Posted by Reel.Deel

Edit: can someone test the 2x_LD-Anime_Skr_v1.0.pth model? It crashes on me.

Didn't crash for me, but the GPU got high rolling for a few seconds... scared.

Does anyone know of a good inpaint model? I only know 1x_NMKD-YandereInpaint_375000_G and 1x_sudo_inpaint_PartialConv2D_424000_G but not sure which one might be better for opaque logos on live action content.

1x_JPEG_60-80.pth can also be good for deblocking old codec sources, need to compare with 1x-Kim2091_DeJpeg.onnx.

[poisondeathray]

Originally Posted by Dogway

Does anyone know of a good inpaint model? I only know 1x_NMKD-YandereInpaint_375000_G and 1x_sudo_inpaint_PartialConv2D_424000_G but not sure which one might be better for opaque logos on live action content.

There are some inpainting algos that use different architectures (cannot be run in avs/vpy/cupscale)

https://github.com/geekyutao/Image-Inpainting
https://github.com/zengyh1900/Awesome-Image-Inpainting

The problem I found with single image/spatial inpainting algos, is the flicker and temporal inconsistencies (you'd have to apply some temporal smoothing after)

Many of them are difficult to get working (at least for me, on Windows) . An interesting one that I actually got to work, and is video inpainting (temporal +/- 3 frames) is OPN . Worked ok for some sources, not for others. YMMV
https://github.com/seoungwugoh/opn-demo


Chainner lists support for 2 spatial algos that I haven't tried yet; but they should "work" since the GUI has direct support

Lama
https://github.com/advimman/lama

MAT
https://github.com/fenglinglwb/MAT

Originally Posted by anton_foy

Tried to find any model for this kind of motion compensation but without success. One would think there would be some out there? Although I do not specifically know what words to use for searching. I tried "model pytorch motion compensation" but mostly got hits about object tracking and similar.

Object tracking is part of it. Also propogation and alignment.

There are typically catergorized into single image vs. video . "Video" algorithms takes >1 frame into account. All the ESRGAN / Real-ESRGAN derivatives are single image only.

Examples of "video" algorithms that use "propogation", "alignment" at runtime are VRT, RVRT, BasicVSR++, RealBasicVSR. At runtime is important because feature are matched in adjacent frames in the specific input video.

Beware video algorithms take lots of memory. Some of them have "tiling" options built in, but the results can be worse especially for operations like deblurring. The whole frame in +/- n frames should be compared on some tasks, otherwise you get discontinunities at the tile boundaries

I posted some examples of BasicVSR/BasicVSR++ in a few threads here and other forums. The provided model works on "textbook" clean cases. A big problem for is degradation step wasn't included , so it's not as generalizable to real world, compression cases. RealBasicVSR adds degradation, but the results are quite blurry / detail loss

[DTL]

Originally Posted by poisondeathray

Examples of "video" algorithms that use "propogation", "alignment" at runtime are VRT, RVRT, BasicVSR++, RealBasicVSR. At runtime is important because feature are matched in adjacent frames in the specific input video.

Beware video algorithms take lots of memory. Some of them have "tiling" options built in, but the results can be worse especially for operations like deblurring. The whole frame in +/- n frames should be compared on some tasks, otherwise you get discontinunities at the tile boundaries

And what is the current RIFE is based at ? Or it is separated engine for doing motion interpolation ? Some wiki https://www.svp-team.com/wiki/RIFE_AI_interpolation says it is based on IFNet neural network.
Also it shows TensorRT at NVIDIA cards can run RIFE about 2x faster in compare with ncnn/Vulkan implementation. So RIFE (IFNet ?) can be compiled (?) to different hardware APIs ?

As for only storing frames in memory - the 1920x1080 frame in 12bytes per sample RGB 32bit float is about 24 MB only. So 1/4 of 1GB onboard RAM can keep 10 frames and it is very good already to study how motion is going between several frames.

I post an issue on some RIFE development project at github about possible changing RIFE algorithm for better denoise process - https://github.com/megvii-research/E...IFE/issues/304 but still no one reply. May be it require to search someone with understanding how RIFE work so the required changes can be implemented.

[Dogway]

Originally Posted by poisondeathray

Many of them are difficult to get working (at least for me, on Windows) . An interesting one that I actually got to work, and is video inpainting (temporal +/- 3 frames) is OPN . Worked ok for some sources, not for others. YMMV
https://github.com/seoungwugoh/opn-demo

Thanks a lot, yes I wonder why temporal information isn't used more frequently in inpainting. I mean it has huge potential not only in restoration but also VFX. I will give it a stab when I'm in Win10.

[poisondeathray]

Originally Posted by Reel.Deel

Edit: can someone test the 2x_LD-Anime_Skr_v1.0.pth model? It crashes on me.

Could be GPU memory ?

Did you try the tiling options ?

- overlap_w, overlap_h\
Overlap width and overlap height of the tiles, respectively.\
Must be less than or equal to `tilesize_w` / `tilesize_h` `/` `2`.\
Default: 0.

- tilesize_w, tilesize_h\
Tile width and height, respectively.\
Use smaller value to reduce GPU memory usage.\
Must be specified when `overlap_w` / `overlap_h` > 0.\
Default: input_width, input_height.

And I tried a few ONNX converted models , they were bit identical to the pytorch version so far...

[poisondeathray]

Originally Posted by poisondeathray

And I tried a few ONNX converted models , they were bit identical to the pytorch version so far...

Ok, some very minor differences (only detectable on amplified differences) between avs, chainner using onnx and vs. pytorch in some cases, when settings are the same (e.g. tiles) . I think some of them might be due to the way it's converted to float or perhaps [0,1] clipping. The differences are so minor they are nothing to worry about

[Reel.Deel]

Originally Posted by poisondeathray

Ok, some very minor differences (only detectable on amplified differences) between avs, chainner using onnx and vs. pytorch in some cases, when settings are the same (e.g. tiles) . I think some of them might be due to the way it's converted to float or perhaps [0,1] clipping. The differences are so minor they are nothing to worry about

Thanks for testing, good to know that there are no big differences. That's the consensus I got from other places. When converting some models I get a warning "Backend: WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function." but from what I read, they are harmless.

Originally Posted by poisondeathray

Could be GPU memory ?

Did you try the tiling options ?

Probably, I need to experiment with those options.

Originally Posted by Dogway

Does anyone know of a good inpaint model? I only know 1x_NMKD-YandereInpaint_375000_G and 1x_sudo_inpaint_PartialConv2D_424000_G but not sure which one might be better for opaque logos on live action content.

1x_JPEG_60-80.pth can also be good for deblocking old codec sources, need to compare with 1x-Kim2091_DeJpeg.onnx.

I'll add those to the package.

Originally Posted by DonCanjas

Would like to see

• 4x_eula_digimanga_bw_v2_nc1_307k
• MangaJPEG
• 1x_GainRESV3 (Aggro,Natural,Passive)
• 1x_DitherDeleter-Smooth_104000_G

Will do.

[Reel.Deel]

Originally Posted by poisondeathray

Could be GPU memory ?

Did you try the tiling options ?

I was able to get it running using those options, although at a snails pace. I guess my GPU is not adequate for the bigger models (1050 now, I plan on upgrading to a 2070 in the upcoming months).

[kedautinh12]

Originally Posted by poisondeathray

I mention it because sometimes an earlier checkpoint produces better results. Here are all the checkpoints for DeGif that I have. Credits to NMKD (n00mkrad)
https://www.mediafire.com/file/2jzoq...+DeGIF.7z/file

Just curious :
1) how long does it take to convert ?
2) are results bit identical to the pytorch version?

can you convert it to onnx??

[poisondeathray]

Originally Posted by kedautinh12

can you convert it to onnx??

https://www.mediafire.com/file/51mae...G_onnx.7z/file

[Arx1meD]

This plugin works on GPU only?