vs-basicvsr

[Selur]

Will do.
Main issue I have atm. with it, is that when the head covers, most of the image, only parts of the head seem to be processed.
This kind of ruins the image if the person has longer hair.

(using detector = 1, doesn't help either)
https://imgsli.com/MTY2MjEw Suzie with weight=1

[ChaosKing]

At least the face now looks like it belongs to the same person

[Selur]

Here's an example of weight 0 and weight 1: https://imgsli.com/MTY2Mjc4

[Selur]

Did some further testing and if upscale is > 1 (weight=1) and the upscaling is later undone it basically can be used as a 'face detail enhancer' which might be interesting.
Also in combination with FeMaSR (with weight<1) it can be used to fix the face distortions FeMaSR does.

This does not fix the issue with half enhanced heads, but seems to be a good way to use the filter.
SwinIRx4+vsCodeFormer: https://imgsli.com/MTY2NDA3, looks nice frame-by-frame, but isn't really temporally stable => adding 3xQTGMC_type3.

Cu Selur

Ps.: got a nice example of what weight 0 can do: https://imgsli.com/MTY2NDI2,...

[Selur]

Okay, small update in regard to Vapoursynth R62:
TensorRT doesn't support Python 3.11 atm.
Latest TensorRT-package (TensorRT-8.6.0.12.Windows10.x86_64.cuda-11.8.zip) from https://developer.nvidia.com/tensorrt does not contain (TensorRT-8.6.0.12.Windows10.x86_64.cuda-11.8.zip\TensorRT-8.6.0.12\python) a cp311 file.
So all the filters that use TensorRT won't work.
=> Hoping NVIDIA will release a Python 3.11 file in the near (who knows) future.

Cu Selur

[Selur]

Maybe interesting to some:
https://github.com/HolyWu/vs-grlir uses real-world image super-resolution model of GRL for Image Restoration
https://imgsli.com/MTcxMDA1

Cu Selur

[ReinerSchweinlin]

Thanx for all the info.. I think its about time to check out your extended version of hybrid, including all the nice "AI" stuff not included in the normal release

[Selur]

@RainerSchweinlin: Send you a pm with links to my current dev + torchAddon.

[ReinerSchweinlin]

Originally Posted by Selur

@RainerSchweinlin: Send you a pm with links to my current dev + torchAddon.

Merci

[Selur]

btw. vs-grlir just got updated and not supports multiple models, cuda_graphs, fp16.

 :param model:           Model to use.
                             0 = Blind Image SR
                             1 = Defocus Deblurring
                             2 = Motion Deblurring (GoPro)
                             3 = Motion Deblurring (RealBlur-J)
                             4 = Motion Deblurring (RealBlur-R)
                             5 = Demosaicking
                             6 = Denoising (sigma 15)
                             7 = Denoising (sigma 25)
                             8 = Denoising (sigma 50)
                             9 = JPEG compression artifact removal (quality 10)
                            10 = JPEG compression artifact removal (quality 20)
                            11 = JPEG compression artifact removal (quality 30)
                            12 = JPEG compression artifact removal (quality 40)
                            13 = Classical Image SR (scale 2)
                            14 = Classical Image SR (scale 3)
                            15 = Classical Image SR (scale 4)

source: https://github.com/HolyWu/vs-grlir/b...ir/__init__.py
Models 0, 12-15 are for upscaling, the others are for filtering.
Nice!

Cu Selur

[Selur]

Has anyone tried https://github.com/tongyuantongyu/vs-NNVISR/ ?

[Selur]

Tried to test it by extracing everything into "F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR" ending with:

F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublasLt64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudart64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvblas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_builder_resource.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_dispatch.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_lean.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_vc_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvonnxparser.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc-builtins64_121.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc64_120_0.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\vs-nnvisr.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\fe_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\fe_n4_4x4_l1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\ff_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\ff_n4_4x4_l1.onnx

Problem is when i use:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core

# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll")

# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)

# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive


clip = core.nnvisr.Super(clip, scale_factor=2, use_fp16=True, model="fe_n2_2x2_l4_yuv1-1.onnx", model_path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR")



# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

I get:

Failed to evaluate the script:
Python exception: Failed to load F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll. GetLastError() returned 1114.

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 3115, in vapoursynth._vpy_evaluate
File "src\cython\vapoursynth.pyx", line 3116, in vapoursynth._vpy_evaluate
File "C:\Users\Selur\Desktop\test_1.vpy", line 8, in 
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll")
File "src\cython\vapoursynth.pyx", line 2847, in vapoursynth.Function.__call__
vapoursynth.Error: Failed to load F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll. GetLastError() returned 1114.

So the loading of the dll already fails.

-> does anyone have an idea what I'm missing/overlooking?

Cu Selur

[Selur]

Got it working.
dll folder needs to explicitly loaded:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import site
import os
import ctypes
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/../vsfilters/ResizeFilter/NNVISR'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]

and Primaries, Transfer and Matrix all have to be BT709 (for RGB only Primaries and Transfer have to be bt709).

Cu Selur
Ps.: the explicit loading is fixed in 'v1.0.0-pre3'

[ReinerSchweinlin]

Cool you got it working

If I understand correctly, this enables multi-frame aware networks to be run, not only single-frame ones. Correct?

[Selur]

Yes, vs-NNVISR support models which use mutliple frames for their calculation, main problem is finding such models (or training them yourself).

We are not aiming at being an alternative of vs-mlrt, but focus mainly on models that do multiple frames to multiple frames enhancement, which vs-mlrt currently not able to do, or at least not easily.

source: issue #1

Side note: VSGAN also supports EGVSR models, which are 'temporal aware'.

Current available models only run with specific values.
"'scale_factor': 2, 'input_count': 2,..., 'interpolation': True" means the model always does scale by 2, takes two frames as input and interpolates the frames by 2.
Also, the YUV420 1/1/1 means that it only works if color fromat is YUV420(P8), _Transfer, _Primaries and _Matrix are bt709.
see: https://github.com/tongyuantongyu/vs...docs/models.md

So to me, it's nice to see something different is being developed, but atm. I won't use it since it seems too much of a hassle.

Cu Selur

Ps.: Seems like the 'Enhance Everything' folks are revamping https://upscale.wiki/wiki/Model_Database into https://openmodeldb.info/

[mastrboy]

Originally Posted by Selur

Ps.: Seems like the 'Enhance Everything' folks are revamping https://upscale.wiki/wiki/Model_Database into https://openmodeldb.info/

Looks good, the wiki format doesn't really work that well for content like that...

vsbasicvsrpp does temporal I think, with the length parameter: https://github.com/HolyWu/vs-basicvs..._init__.py#L50

[Selur]

Yes, BasicVSR++ does, but most models you load in VSGAN/vsMLRT do not.