vs-basicvsr

[ReinerSchweinlin]

Sorry, I can´t help, thats beyond my capabilities

But I just wanted to say "thanks" for all you do !!

[Selur]

btw. did anyone test NNVISR ? (https://github.com/tongyuantongyu/vs-NNVISR)

[lansing]

Originally Posted by Selur

btw. did anyone test NNVISR ? (https://github.com/tongyuantongyu/vs-NNVISR)

I couldn't get it working on vsedit2 or vd2 or vsedit_mod. I manual install the plugin and its dependencies. Somehow the plugin won't auto load when I put it in my roaming folder, so I'll have to put everything in Vapoursynth's installation plugin folder. But everything just crashed after some loading time.

import vapoursynth as vs

core = vs.core

clip = core.bs.VideoSource(r'E:\video samples\hand wave.mkv')

model_config = {'scale_factor': 2, 'input_count': 2, 'feature_count': 64, 'extraction_layers': 4, 'interpolation': True, 'extra_frame': True, 'double_frame': True, 'model': 'cycmunet/vimeo90k-deblur'}

clip = core.nnvisr.Super(clip,
                         model_path=r"C:\Program Files\VapourSynth\plugins",  **model_config)

clip.set_output()

[Selur]

When I first tested this, I used this file structure

F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublasLt64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudart64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvblas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_builder_resource.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_dispatch.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_lean.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_vc_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvonnxparser.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc-builtins64_121.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc64_120_0.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\test_1.vpy
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\timing.cache
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\vs-nnvisr.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\timing.cache
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur\yuv_1_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur\yuv_1_1_1\fe_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur\yuv_1_1_1\ff_n2a+_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur\yuv_1_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur\yuv_1_1_1\fe_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur\yuv_1_1_1\ff_n2a+_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur\yuv_1_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur\yuv_1_1_1\fe_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur\yuv_1_1_1\ff_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k\rgb_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k\rgb_1_1\fe_n4_4x4_l1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k\rgb_1_1\ff_n4_4x4_l1.onnx

and the following script:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import site
import os
import ctypes
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/../vsfilters/ResizeFilter/NNVISR'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]


# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll")

# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)

# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive

clip = core.resize.Bicubic(clip=clip, matrix_in_s="470bg", matrix_s="709", range_in=0, range=0) # converting to bt709 for NNVISR
clip = core.std.SetFrameProps(clip, _Transfer=1)
clip = core.std.SetFrameProps(clip, _Primaries=1)

model_config = {'scale_factor': 2, 'input_count': 2, 'feature_count': 64, 'extraction_layers': 4, 'interpolation': True, 'extra_frame': True, 'double_frame': True, 'model': 'cycmunet/vimeo90k-deblur'}
clip = core.nnvisr.Super(clip, model_path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR", **model_config)

# Output
clip.set_output()

in a portable Vapoursynth setup. But I had my struggles too (https://github.com/tongyuantongyu/vs-NNVISR/issues/2).
Maybe this helps you getting it working.

Cu Selur

Ps.: iirc. it took ages to get engine files created on my setup when I tested it.

[julius666]

Does anyone know what happened to vs-femasr? The github repository is not available anymore.

[kedautinh12]

HolyWu was deleted it (maybe)

[Selur]

Yup, like a few other repos HolyWu deleted it. (or set it private)

[julius666]

Originally Posted by Selur

Yup, like a few other repos HolyWu deleted it. (or set it private)

I thought so but do we have any information regarding why he did that? It was one of the best (if not THE best) upscalers in vapoursynth.

[Selur]

No (at least not that I know of), you would have to ask him (maybe by posting an issue in one of this other projects https://github.com/HolyWu/).
Probably because there hasn't been an update to it, not much feedback and keeping it working with latest pytorch&co was too much trouble.

[Selur]

btw. for those who need colorization of b&w content https://github.com/HolyWu/vs-ddcolor might be worth a try:

did some small tests which do seem good so far.

Cu Selur

[kedautinh12]

Wow, respect HolyWu

[Selur]

https://github.com/piddnad/DDColor should get the main respect, HolyWu 'just' wrote a Vapoursynth Wrapper for it.

[kedautinh12]

Ok, piddnad too

[Selur]

Anyone has an idea how to stabilize the colorization output of ddcolor? (see: https://forum.selur.net/thread-3532.html)

[Selur]

https://github.com/dan64/vs-deoldify might be interesting for those interested in colorization

[ReinerSchweinlin]

Thank you.

I have a somewhat amateur question:

If this is based on pytorch - what relevance does the used GPU have in this case? Some hybrid filters are clearly based on specific stuff like vulkan or CUDA (CDNN) or some other filters use OPENCL... How does it work here?

I have been playing around with some StableDiffusion GUIs and here It seems (as far as I understand it), all that needs to be modified is how Pytorch passes the calls from the GUI to the underlying inferences. So if I want to use DirectML, I need the py-directML stuff, if i have Nvidia, I stick to the tensorRT or CUDA, hence must install pytor...cuda... etc... So basically, as long as some backend in pytorch for my specific hardware exists, the filter itself will run...

Could you help straighten out my somewhat fuzzy thinking of how it is related ?

Thanx

[Selur]

afaik vs-deoldify uses cudnn as backend (see: https://github.com/dan64/vs-deoldify...ify/_device.py)

[ReinerSchweinlin]

Originally Posted by Selur

afaik vs-deoldify uses cudnn as backend (see: https://github.com/dan64/vs-deoldify...ify/_device.py)

Thanx for pointing me in the right direction

[Selur]

Side note for those using tensorrt: atm. there isn't a Python 3.12 version of the bindings (tensorrt-8.6.1-cp311-none-win_amd64.whl is the latest currently), so switching to Vapoursynth R66 will not work.

Cu Selur

[PatchWorKs]

Just out of curiosity: does anyone tested vs-mlrt performances on Intel CPU+iGPU ?