vs-basicvsr

[Selur]

Will do.
Main issue I have atm. with it, is that when the head covers, most of the image, only parts of the head seem to be processed.
This kind of ruins the image if the person has longer hair.

(using detector = 1, doesn't help either)
https://imgsli.com/MTY2MjEw Suzie with weight=1

[ChaosKing]

At least the face now looks like it belongs to the same person

[Selur]

Here's an example of weight 0 and weight 1: https://imgsli.com/MTY2Mjc4

[Selur]

Did some further testing and if upscale is > 1 (weight=1) and the upscaling is later undone it basically can be used as a 'face detail enhancer' which might be interesting.
Also in combination with FeMaSR (with weight<1) it can be used to fix the face distortions FeMaSR does.

This does not fix the issue with half enhanced heads, but seems to be a good way to use the filter.
SwinIRx4+vsCodeFormer: https://imgsli.com/MTY2NDA3, looks nice frame-by-frame, but isn't really temporally stable => adding 3xQTGMC_type3.

Cu Selur

Ps.: got a nice example of what weight 0 can do: https://imgsli.com/MTY2NDI2,...

[Selur]

Okay, small update in regard to Vapoursynth R62:
TensorRT doesn't support Python 3.11 atm.
Latest TensorRT-package (TensorRT-8.6.0.12.Windows10.x86_64.cuda-11.8.zip) from https://developer.nvidia.com/tensorrt does not contain (TensorRT-8.6.0.12.Windows10.x86_64.cuda-11.8.zip\TensorRT-8.6.0.12\python) a cp311 file.
So all the filters that use TensorRT won't work.
=> Hoping NVIDIA will release a Python 3.11 file in the near (who knows) future.

Cu Selur

[Selur]

Maybe interesting to some:
https://github.com/HolyWu/vs-grlir uses real-world image super-resolution model of GRL for Image Restoration
https://imgsli.com/MTcxMDA1

Cu Selur

[ReinerSchweinlin]

Thanx for all the info.. I think its about time to check out your extended version of hybrid, including all the nice "AI" stuff not included in the normal release

[Selur]

@RainerSchweinlin: Send you a pm with links to my current dev + torchAddon.

[ReinerSchweinlin]

Originally Posted by Selur

@RainerSchweinlin: Send you a pm with links to my current dev + torchAddon.

Merci

[Selur]

btw. vs-grlir just got updated and not supports multiple models, cuda_graphs, fp16.

 :param model:           Model to use.
                             0 = Blind Image SR
                             1 = Defocus Deblurring
                             2 = Motion Deblurring (GoPro)
                             3 = Motion Deblurring (RealBlur-J)
                             4 = Motion Deblurring (RealBlur-R)
                             5 = Demosaicking
                             6 = Denoising (sigma 15)
                             7 = Denoising (sigma 25)
                             8 = Denoising (sigma 50)
                             9 = JPEG compression artifact removal (quality 10)
                            10 = JPEG compression artifact removal (quality 20)
                            11 = JPEG compression artifact removal (quality 30)
                            12 = JPEG compression artifact removal (quality 40)
                            13 = Classical Image SR (scale 2)
                            14 = Classical Image SR (scale 3)
                            15 = Classical Image SR (scale 4)

source: https://github.com/HolyWu/vs-grlir/b...ir/__init__.py
Models 0, 12-15 are for upscaling, the others are for filtering.
Nice!

Cu Selur

[Selur]

Has anyone tried https://github.com/tongyuantongyu/vs-NNVISR/ ?

[Selur]

Tried to test it by extracing everything into "F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR" ending with:

F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublasLt64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudart64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvblas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_builder_resource.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_dispatch.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_lean.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_vc_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvonnxparser.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc-builtins64_121.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc64_120_0.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\vs-nnvisr.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\fe_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\fe_n4_4x4_l1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\ff_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\ff_n4_4x4_l1.onnx

Problem is when i use:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core

# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll")

# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)

# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive


clip = core.nnvisr.Super(clip, scale_factor=2, use_fp16=True, model="fe_n2_2x2_l4_yuv1-1.onnx", model_path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR")



# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

I get:

Failed to evaluate the script:
Python exception: Failed to load F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll. GetLastError() returned 1114.

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 3115, in vapoursynth._vpy_evaluate
File "src\cython\vapoursynth.pyx", line 3116, in vapoursynth._vpy_evaluate
File "C:\Users\Selur\Desktop\test_1.vpy", line 8, in 
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll")
File "src\cython\vapoursynth.pyx", line 2847, in vapoursynth.Function.__call__
vapoursynth.Error: Failed to load F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll. GetLastError() returned 1114.

So the loading of the dll already fails.

-> does anyone have an idea what I'm missing/overlooking?

Cu Selur

[Selur]

Got it working.
dll folder needs to explicitly loaded:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import site
import os
import ctypes
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/../vsfilters/ResizeFilter/NNVISR'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]

and Primaries, Transfer and Matrix all have to be BT709 (for RGB only Primaries and Transfer have to be bt709).

Cu Selur
Ps.: the explicit loading is fixed in 'v1.0.0-pre3'

[ReinerSchweinlin]

Cool you got it working

If I understand correctly, this enables multi-frame aware networks to be run, not only single-frame ones. Correct?

[Selur]

Yes, vs-NNVISR support models which use mutliple frames for their calculation, main problem is finding such models (or training them yourself).

We are not aiming at being an alternative of vs-mlrt, but focus mainly on models that do multiple frames to multiple frames enhancement, which vs-mlrt currently not able to do, or at least not easily.

source: issue #1

Side note: VSGAN also supports EGVSR models, which are 'temporal aware'.

Current available models only run with specific values.
"'scale_factor': 2, 'input_count': 2,..., 'interpolation': True" means the model always does scale by 2, takes two frames as input and interpolates the frames by 2.
Also, the YUV420 1/1/1 means that it only works if color fromat is YUV420(P8), _Transfer, _Primaries and _Matrix are bt709.
see: https://github.com/tongyuantongyu/vs...docs/models.md

So to me, it's nice to see something different is being developed, but atm. I won't use it since it seems too much of a hassle.

Cu Selur

Ps.: Seems like the 'Enhance Everything' folks are revamping https://upscale.wiki/wiki/Model_Database into https://openmodeldb.info/

[mastrboy]

Originally Posted by Selur

Ps.: Seems like the 'Enhance Everything' folks are revamping https://upscale.wiki/wiki/Model_Database into https://openmodeldb.info/

Looks good, the wiki format doesn't really work that well for content like that...

vsbasicvsrpp does temporal I think, with the length parameter: https://github.com/HolyWu/vs-basicvs..._init__.py#L50

[Selur]

Yes, BasicVSR++ does, but most models you load in VSGAN/vsMLRT do not.

[Selur]

Trying to set up a new portable Vapoursynth R64 with BasicVSR++&Co I'm stuck.
What I did so far to create a portable Vapoursynth with pytorch&co:

• created a new empty Vapoursynth-folder
• downloaded 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3111/
• extracted the Python download into the download into the 'Vapoursynth'-folder
• downloaded 'VapourSynth64-Portable-R64' from https://github.com/vapoursynth/vapoursynth/releases
• extracted the Vapoursynth portable download into the 'Vapoursynth'-folder
• downloaded get-pip.py from https://bootstrap.pypa.io/get-pip.py and save it into the 'Vapoursynth'-folder
• opened a 'Windows Command Prompt'-window and navigate into the 'Vapoursynth'-folder
• installed pip by calling :
  

  python get-pip.py

• opened the python311._pth in a text addition and added the following to lines above anything else in that file and saved the file
  

  Scripts
  Lib\site-packages

• installed pytorch using:
  

  python -m pip install torch===2.1.0+cu121 torchvision===0.16.0 -f https://download.pytorch.org/whl/torch_stable.html

• installed VSGAN (https://github.com/rlaPHOENiX/VSGAN)
  

  python -m pip install vsgan

• installed DPIR (https://github.com/HolyWu/vs-dpir)
  • downloaded https://github.com/HolyWu/vs-rife/re...6.1.6_win64.7z extracted tensorrt-8.6.1-cp311-none-win_amd64.whl file into the Vapoursynth-folder.
  • extracted the other dlls into a 'torch_dependencies/bin'-folder inside the Vapoursynth folder
  • called

    set PATH=<PATH TO torch_dependencies>\bin;%PATH%

  • installed vsdir and its dependencies
    

    python -m pip install tensorrt-8.6.1-cp311-none-win_amd64.whl
    python -m pip install -U vsdpir
    python -m pip install packaging
    python -m vsdpir

• installed Real-ESGRAN (https://github.com/HolyWu/vs-realesrgan)
  

  python -m pip install -U vsrealesrgan
  python -m vsrealesrgan

• installed vs-rife (https://github.com/HolyWu/vs-rife)
  

  python -m pip install --upgrade vsrife
  python -m vsrife

• installed GMFSS (https://github.com/HolyWu/vs-gmfss_fortuna)
  

  python -m pip install -U vsgmfss-fortuna

• installed SwinIR (https://github.com/HolyWu/vs-swinir)
  

  python -m pip install -U vsswinir
  python -m vsswinir

• installe HINet (https://github.com/HolyWu/vs-hinet)
  

  python -m pip install --upgrade vshinet
  python -m vshinet

• installed AnimeSR (https://github.com/HolyWu/vs-animesr)
  

  python -m pip install -U vsanimesr

• installed FeMaSR (https://github.com/HolyWu/vs-femasr)
  

  python -m pip install -U vsfemasr
  python -m vsfemasr

• installed SCUNet (https://github.com/HolyWu/vs-scunet)
  

  python -m pip install -U vsscunet
  python -m vsscunet

• installed CodeFormer (https://github.com/HolyWu/vs-codeformer)
  

  python -m pip install -U vscodeformer
  python -m vscodeformer

• installed GRLIR (https://github.com/HolyWu/vs-grlir)
  

  python -m pip install -U vsgrlir
  python -m vsgrlir

Not I wanted to call:

python -m pip install -U openmim
mim install "mmcv>=2.0.0"
python -m pip install -U vsbasicvsrpp
python -m vsbasicvsrpp

to install BasicVSR++, problem is when calling:

python -m pip install -U openmim

it fails with:

Collecting aliyun-python-sdk-core>=2.13.12 (from oss2~=2.17.0->openxlab->opendatalab->openmim)
  Using cached aliyun-python-sdk-core-2.14.0.tar.gz (443 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\Selur\AppData\Local\Temp\pip-install-t2r9eu4j\aliyun-python-sdk-core_b912f61c599e490b859580d9c77cbbd2\setup.py", line 41, in <module>
          VERSION = __import__(PACKAGE).__version__
                    ^^^^^^^^^^^^^^^^^^^
      ModuleNotFoundError: No module named 'aliyunsdkcore'
      [end of output]

see: https://github.com/HolyWu/vs-basicvsrpp/issues/37

I also tried installing aliyun-python-sdk-core
with:

python -m pip install aliyunsdkcore

and

python -m pip install aliyun-python-sdk-core-v3

but both calls end the same way, with the same error I got above.

=> does anyone have an idea how to fix this?

Cu Selur

[Selur]

Seems to be related to the openmim version, when using:

python -m pip install -U openmim==0.3.7

(0.3.7 was the version I had installed in a previous portable setup)
openmim does install without an issue, but then

set PATH=f:\Hybrid\64bit\Vapoursynth\Scripts;%PATH%
mim install "mmcv>=2.0.0"

fails with:

Looking in links: https://download.openmmlab.com/mmcv/dist/cu121/torch2.1.0/index.html
Collecting mmcv>=2.0.0
  Using cached mmcv-2.0.1.tar.gz (469 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      C:\Users\Selur\AppData\Local\Temp\pip-install-xs23k5n_\mmcv_a93dd8751cad4bf1bafcae67f7e00b43\setup.py:5: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
        from pkg_resources import DistributionNotFound, get_distribution, parse_version
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\Selur\AppData\Local\Temp\pip-install-xs23k5n_\mmcv_a93dd8751cad4bf1bafcae67f7e00b43\setup.py", line 466, in <module>
          ext_modules=get_extensions(),
                      ^^^^^^^^^^^^^^^^
        File "C:\Users\Selur\AppData\Local\Temp\pip-install-xs23k5n_\mmcv_a93dd8751cad4bf1bafcae67f7e00b43\setup.py", line 423, in get_extensions
          ext_ops = extension(
                    ^^^^^^^^^^
        File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py", line 1076, in CUDAExtension
          library_dirs += library_paths(cuda=True)
                          ^^^^^^^^^^^^^^^^^^^^^^^^
        File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py", line 1210, in library_paths
          paths.append(_join_cuda_home(lib_dir))
                       ^^^^^^^^^^^^^^^^^^^^^^^^
        File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py", line 2416, in _join_cuda_home
          raise OSError('CUDA_HOME environment variable is not set. '
      OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "f:\Hybrid\64bit\Vapoursynth\Scripts\mim.exe\__main__.py", line 7, in <module>
  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\mim\commands\install.py", line 73, in cli
    exit(exit_code)
    ^^^^
NameError: name 'exit' is not defined

¯\(°_o)/¯
(same when using openmim==0.3.8, openmim==0.3.9 fails at the install of openmim)

Cu Selur

[Selur]

Pre-built package for PyTorch 2.1+cu121 is probably not available. File an issue to MMCV if you really hurry to have it.

source: https://github.com/HolyWu/vs-basicvs...ent-1749183229
=> so there's hope, but will have to wait