vs-basicvsr

[Selur]

The interessting part are the screenshots, so the language shouldn't be an issue. Not much text so, Google Translate or similar should work too.

Cu Selur

[poisondeathray]

Originally Posted by Selur

Btw. has anyone an example where BasicVSR show some impressive results when starting from a 480p or 576p source?
Normal resizing using i.e. Lanczos and adding some contrast sharpening seems to produce more impressive results.
Only thing that I found is that it does do well is upscaling QCIF content, and I wonder whether that is it's goal.

BasicVSR works ok in simular situations to where the ESRGAN base model works ok - ie. fairly clean, downsampled sources - really an oversampled SD source... Which is NOT what people typically have - it's usually some noisy, soft DVD source, or SD crappy home video that people want to upscale. But those clean datasets are what most publically available algorithms were trained on: Clean HR ground truth source, downscaled by 4. Usually with bicubic. BasicVSR used selections from the Reds4 and Vimeo90k datasets. qcif "suzie" is really an oversampled source. If you take some ok quality UHD, and downscale it, that will work too. What's needed is training of specific models that take account for noise, degredation, compression artifacts like mpeg2 artifacts, etc... stuff like bad YT compression and degredation after the downscaling step. There are specific datasets that include noise - but they are synthetic noise - and they weren't used for these default models .

So in those situations where the model is applicable - BasicVSR is going to produce more detailed images than , say lanczos3 + any form of sharpen, but less detailed results than default ESRGAN or ESRGAN+ models on single image examination. On single image, ESRGAN+ is going to look more "wow" in applicable source situations. The main difference is BasicVSR is more consistent temporally, when you look at actual video, or more than 1 image. It resembles tecogan in that regard, but produces better results detail wise than the default tecogan model (which isn't the same as the tecogan published paper model). For all single image SR approaches you often need to use some type of temporal smoothing to make it decent for video purposes (and that smoothing reduces the fine details too) , otherwise you are prone to flickering aliasing artifacts, objects changing shapes (as single image prediction is slightly different on different frames) .

So the question is can you use something like ESRGAN, add some QTGMC in progressive mode - to get similar temporally consistent results to BasicVSR, but hopefully keep more fine details? The answer is generally no. The artifacts are too "baked" in - that's the strength of basicvsr

[poisondeathray]

Some apng comparisons. They should animate and loop in most browsers

1:1 non-AR corrected crops, unfiltered. 720x480 => 2880x1920.

Note the temporal flickering artifacts (aliasing flicker, and prediction flicker).

You can see the earrings look like random lightning in the ESRGAN+ results.



No amount of sharpening will transform lanczos3 into as detailed as BasicVSR. No amount of temporal filtering will transform lanczos3, or ESRGAN+ results into as temporally consistent results as BasicVSR




The license plate, you can make out the numbers/letters with lanczos, but it's much more clear in the BasicVSR. It doesn't take a stretch of the imagination that slightly smaller or a more distant plate might be illegible when processed with lanczos , but partially readable by BasicVSR



ESRGAN and psnr model mess up letters/numbers/text (many GAN, NN models do), I won't include in the animated temporal demo, but here is a still image compare animated demo. ESRGAN default does good on BG foliage textures, but messes up text. None of the GAN scalers have "fuzzy" lines or bad aliasing - that's what they excel at. Early gen NNEDI/1/2/3 is NN (neural net) based, but signficantly worse in terms of aliasing artifacts for upscaling 4x than modern GAN scalers





But these are cleanly downsample source examples, much like the training set. ie. They don't have MPEG2 macroblocks or or grain plate added on top like you might with some film DVD

Some trained degraded jpeg ESRGAN models work ok for single image jpeg interpolation - I would like to see more BasicVSR models that utilize some typical degraded aspects of common video

[lansing]

The car plate image is amazing

[poisondeathray]

Keep in mind those are "textbook" clean sources . Normal dashcam / security cam footage would have a bunch of data loss, compression artifacts. The training set for the 3 base models do not account for that

There is an improved version, BasicVSR++ , but the code has not been released yet . I hope HolyWu brings it to vapoursynth when it's released

[Selur]

@poisondeathray: Thanks for the explaination, that explains why I didn't get and impressive results from my usual content.

[Selur]

There is an improved version, BasicVSR++ ,..

Looks impressive: https://ckkelvinchan.github.io/projects/BasicVSR++/

I hope HolyWu brings it to vapoursynth when it's released

fingers crossed

Cu Selur

[edcrfv94]

Originally Posted by poisondeathray

Keep in mind those are "textbook" clean sources . Normal dashcam / security cam footage would have a bunch of data loss, compression artifacts. The training set for the 3 base models do not account for that

There is an improved version, BasicVSR++ , but the code has not been released yet . I hope HolyWu brings it to vapoursynth when it's released

released 10 days ago
https://github.com/open-mmlab/mmedit...basicvsr_pp.py

MMEditing v0.10.0 Release
Support LIIF-RDN (CVPR'2021)
Support BasicVSR++ (NTIRE'2021)
https://github.com/open-mmlab/mmedit...es/tag/v0.10.0

[poisondeathray]

Originally Posted by edcrfv94

released 10 days ago
https://github.com/open-mmlab/mmedit...basicvsr_pp.py

MMEditing v0.10.0 Release
Support LIIF-RDN (CVPR'2021)
Support BasicVSR++ (NTIRE'2021)
https://github.com/open-mmlab/mmedit...es/tag/v0.10.0

Nice!!

[aegisofrime]

@poisondeathray: Could I ask what sort of speed are you getting in your test? Upscaling just from 720x480 -> 1280x720 I'm getting like 0.05fps with a Ryzen 5950X + RTX 3060Ti, and I'm wondering if I'm doing something wrong.

[poisondeathray]

Originally Posted by aegisofrime

@poisondeathray: Could I ask what sort of speed are you getting in your test? Upscaling just from 720x480 -> 1280x720 I'm getting like 0.05fps with a Ryzen 5950X + RTX 3060Ti, and I'm wondering if I'm doing something wrong.

Similar speeds - it's very slow.

[gugglu]

Hi folks, i am trying to test the vsbasicVSR but whenever i try to load the BasicVSR it says cuda not available. any hint what i am missing, tia.

script using to test

import vapoursynth as vs
import numpy
import torch
from vsbasicvsr import BasicVSR

core = vs.get_core()

src = core.dgdecodenv.DGSource(r'C:\Users\User89\Desktop\test\00000.dgi')
src = core.fmtc.bitdepth(src, bits=32)
src = core.fmtc.resample(src, kernel='spline36', css='444')
src = core.fmtc.matrix(src, mat='709', col_fam=vs.RGB)
ret = BasicVSR(src)

ret.set_output()

I get this error when checking the above script

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2242, in vapoursynth.vpy_evaluateScript
File "src\cython\vapoursynth.pyx", line 2243, in vapoursynth.vpy_evaluateScript
File "C:/Users/User89/Desktop/VapourSynthEditor-r19-64bit/Untitled.vpy", line 12, in 
src = core.fmtc.matrix(src, mat='709', col_fam=vs.RGB)
File "C:\Users\User89\AppData\Local\Programs\Python\Python39\lib\site-packages\vsbasicvsr\__init__.py", line 56, in BasicVSR
raise vs.Error('BasicVSR: CUDA is not available')
vapoursynth.Error: BasicVSR: CUDA is not available

[poisondeathray]

Originally Posted by gugglu

vapoursynth.Error: BasicVSR: CUDA is not available

Drivers ?

Did you install pytorch with cuda ?

Does cuda work ok for you in other applications, or vapoursynth ? eg. try eedi2cuda
https://github.com/AmusementClub/VapourSynth-EEDI2CUDA

[gugglu]

Originally Posted by poisondeathray

Drivers ?

Did you install pytorch with cuda ?

Does cuda work ok for you in other applications, or vapoursynth ? eg. try eedi2cuda
https://github.com/AmusementClub/VapourSynth-EEDI2CUDA

Hi poisondeathray, i've installed cuda (cuda_11.0.2_451.48_win10) but on its own here are some scrn.shots of process please have a look.

Cuda installation:
https://prnt.sc/1qyobff
https://prnt.sc/1qyocvh
https://prnt.sc/1qyodye
https://prnt.sc/1qyp184
https://prnt.sc/1qyp3ly
installed Drivers:
https://prnt.sc/1qyp9ae
and pytorch was installed with vsbasicvsr via pip as instructed on a github .
https://prnt.sc/1qyrk22
yes EEDi2CUDA works fine
https://prnt.sc/1qyrd78

[poisondeathray]

Originally Posted by gugglu

H
and pytorch was installed with vsbasicvsr via pip as instructed on a github .
https://prnt.sc/1qyrk22

You're missing pytorch with cuda . You only have pytorch cpu

There should be a folder torch-xxx-cuxxx

Where "xxx" is the version number, cu is for cuda

If you go here, and select the boxes that apply to you, it will give you command line to install correct version with cuda
https://pytorch.org/get-started/locally/

[Selur]

Also, if you want I can send you a link to the 'addon' for Hybrid I just uploaded, it's basically a portable Vapoursynth&Python install with:

• vsbasicvsr (https://github.com/HolyWu/vs-basicvsr)
• vsdpir (https://github.com/HolyWu/vs-dpir)
• vsffdnet (https://github.com/HolyWu/vs-ffdnet)
• vsgan (https://github.com/rlaPHOENiX/VSGAN) + a bunch of vsgan_models (https://upscale.wiki/wiki/Model_Database)
• vsrealesgran (https://github.com/HolyWu/vs-realesrgan/)
• vsrife (https://github.com/HolyWu/vs-rife)

packed size ~9GB
if you want a link to the download send me a pm.

Cu Selur

[gugglu]

Originally Posted by poisondeathray

You're missing pytorch with cuda . You only have pytorch cpu

There should be a folder torch-xxx-cuxxx

Where "xxx" is the version number, cu is for cuda

If you go here, and select the boxes that apply to you, it will give you command line to install correct version with cuda
https://pytorch.org/get-started/locally/

thanks a lot poisondeathray its working after downloading pytorch with cuda with windows command line from a pytorch link

Microsoft Windows [Version 10.0.19042.1165]
(c) Microsoft Corporation. All rights reserved.

C:\Users\User89\AppData\Local\Programs\Python\Python39\Scripts>pip3 install torch==1.9.0+cu102 torchvision==0.10.0+cu102 torchaudio===0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.9.0+cu102
  Downloading https://download.pytorch.org/whl/cu102/torch-1.9.0%2Bcu102-cp39-cp39-win_amd64.whl (1440.4 MB)
     |████████████████████████████████| 1440.4 MB 2.8 kB/s
Collecting torchvision==0.10.0+cu102
  Downloading https://download.pytorch.org/whl/cu102/torchvision-0.10.0%2Bcu102-cp39-cp39-win_amd64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 3.3 MB/s
Collecting torchaudio===0.9.0
  Using cached torchaudio-0.9.0-cp39-cp39-win_amd64.whl (215 kB)
Requirement already satisfied: typing-extensions in c:\users\user89\appdata\local\programs\python\python39\lib\site-packages (from torch==1.9.0+cu102) (3.10.0.2)
Requirement already satisfied: pillow>=5.3.0 in c:\users\user89\appdata\local\programs\python\python39\lib\site-packages (from torchvision==0.10.0+cu102) (8.3.1)
Requirement already satisfied: numpy in c:\users\user89\appdata\local\programs\python\python39\lib\site-packages (from torchvision==0.10.0+cu102) (1.21.2)
Installing collected packages: torch, torchvision, torchaudio
  Attempting uninstall: torch
    Found existing installation: torch 1.9.0
    Uninstalling torch-1.9.0:
      Successfully uninstalled torch-1.9.0
  Attempting uninstall: torchvision
    Found existing installation: torchvision 0.10.0
    Uninstalling torchvision-0.10.0:
      Successfully uninstalled torchvision-0.10.0
Successfully installed torch-1.9.0+cu102 torchaudio-0.9.0 torchvision-0.10.0+cu102

[gugglu]

Originally Posted by Selur

Also, if you want I can send you a link to the 'addon' for Hybrid I just uploaded, it's basically a portable Vapoursynth&Python install with:

• vsbasicvsr (https://github.com/HolyWu/vs-basicvsr)
• vsdpir (https://github.com/HolyWu/vs-dpir)
• vsffdnet (https://github.com/HolyWu/vs-ffdnet)
• vsgan (https://github.com/rlaPHOENiX/VSGAN) + a bunch of vsgan_models (https://upscale.wiki/wiki/Model_Database)
• vsrealesgran (https://github.com/HolyWu/vs-realesrgan/)
• vsrife (https://github.com/HolyWu/vs-rife)

packed size ~9GB
if you want a link to the download send me a pm.

Cu Selur

that seems quite interesting, Selur thank you

[poisondeathray]

Thanks HolyWu

vs-BasicVSR++
https://github.com/HolyWu/vs-basicvsrpp

[Selur]

nice!

 Support either x4 upsampling (for model 0-2) or same size output (for model 3-5).
    For model 0-2, input resolution must be at least 64.
    For model 3-5, input resolution must be at least 256 and mod-4.
    Parameters:
        clip: Clip to process. Only planar format with float sample type of 32 bit depth is supported.
        model: Model to use.
            0 = REDS
            1 = Vimeo-90K (BI)
            2 = Vimeo-90K (BD)
            3 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 1
            4 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 2
            5 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 3

Is there a difference between BasicVSR and BasicVSR++ if model 0-2 are used, or is it the same resizing as BasicVSR (model 0-2) and additional models for cleaning?