vs-basicvsr

[Selur]

Also did anyone manage to install this in a portable Vapoursynth environment on Windows?
Calling:

python -m pip install --upgrade vsbasicvsrpp

first failed with

ERROR: Could not find a version that satisfies the requirement vapoursynth==54 (from versions: 39, 40, 41, 42, 43, 44, 45, 46, 47, 47.1, 47.2, 48, 49, 50, 51)
ERROR: No matching distribution found for vapoursynth==54

after renaming the dummy 'VapourSynth-53.dist-info', I created to install VSGAN, to 'VapourSynth-54.dist-info', calling:

python -m pip install --upgrade vsbasicvsrpp

failed with:

OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.

calling:

set CUDA_HOME=I:/Hybrid/64bit/Vapoursynth/Lib/site-packages/torch/cuda

(not toally sure this is correct)
and then:

python -m pip install --upgrade vsbasicvsrpp

it fails with:

I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py:305: UserWarning: Error checking compiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden
      warnings.warn(f'Error checking compiler version for {compiler}: {error}')

I get the same error when calling:

python -m pip install mmcv-full==1.3.12 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.9.0/index.htm

-> that's the point where I gave up, so if anyone figures out how to install vsbasicvsrpp in a protable Vapoursynth environment please let me know.

Cu Selur

[poisondeathray]

Originally Posted by Selur

Is there a difference between BasicVSR and BasicVSR++ if model 0-2 are used, or is it the same resizing as BasicVSR (model 0-2) and additional models for cleaning?

I've only done a few tests so far , but some early observations/comments - basicvsrpp is marginally better with the same model interval size compared to basicvsr. Not a major difference. The default interval size is different, 30 for ++, vs 7*2+1=15

Models 3-5 are from the NTIRE 2021 Quality enhancement of heavily compressed videos Challenge , which take HEVC compressed videos using fixed qp and low bitrate encodings - so those pre-trained models should factor in some compression degredation (at least HEVC type, not necessarily MPEG2, or AVC). It' s nice to see some other types of degradation training and models, but 3 and 5 tend to be very smooth (ie. no detail) . 4 has more detail but more artifacts. Models 3-5 don't upscale

I haven't done enough testing to see if using a much larger interval size helps or hinders in general. It appears a very small interval size is worse. Larger sizes take more memory and are slower

[poisondeathray]

Originally Posted by Selur

-> that's the point where I gave up, so if anyone figures out how to install vsbasicvsrpp in a protable Vapoursynth environment please let me know.

I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py:305: UserWarning: Error checking compiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden
warnings.warn(f'Error checking compiler version for {compiler}: {error}')

Not sure, I used installed environment, but I had problems at first. My errors msg was slightly different - it needed MS Visual C++ compiler to build the "wheels" to install other components. I'm wondering how the compiler is accessed in a "portable" environment ?

[Selur]

I'm wondering how the compiler is accessed in a "portable" environment ?

No clue either, haven't run into the problem before.

Cu Selur

[poisondeathray]

HolyWu added update a few hours ago and made install "easier" on Windows. Maybe try this new one
https://github.com/HolyWu/vs-basicvsrpp

Installing mmcv-full on Windows is a bit complicated as it requires Visual Studio and other tools to compile CUDA ops. So I have uploaded the built file compiled with CUDA 11.1 for Windows users and you can install it by executing the following command.

pip install https://github.com/HolyWu/vs-basicvsrpp/releases/download/v1.0.0/mmcv_full-1.3.12-cp39-cp39-win_amd64.whl

[Selur]

Thanks ! using that call it works for me too.

Cu Selur

[Selur]

Has anyone tried https://github.com/HolyWu/vs-swinir ? (didn't want to create a new thread )
-> man this is too slow on my machine to be useful for normal usage on my gpu (Geforce GTX 1070ti)

[PatchWorKs]

Originally Posted by Selur

Normal resizing using i.e. Lanczos and adding some contrast sharpening seems to produce more impressive results.

HolyWu has just ported - after my request - SwinIR to VS, can someone make some "real world" test with it ?

https://github.com/HolyWu/vs-swinir

Originally Posted by Selur

man this is too slow on my machine to be useful for normal usage on my gpu (Geforce GTX 1070ti)

For individual frames (= pics) you can test it here, but a video-oriented colab notebook like this would be great (I don't own a discrete GPU at all) !


EDIT
Out of curiosity: do you think the new Apple chips (M1 Pro / Max) could speed up operations?

[Selur]

Out of curiosity: do you think the new Apple chips (M1 Pro / Max) could speed up operations?

Without:
a. pytorch support
b. rewriting of the exitistn plugins
-> no

For individual frames (= pics)

I can run it for single pics fine, but I get like 0.005fps for sd->hd on my system, which simply is too slow for me to be usable.

[ReinerSchweinlin]

Originally Posted by Selur

I can run it for single pics fine, but I get like 0.005fps for sd->hd on my system, which simply is too slow for me to be usable.

thanx for testing How much vram did it use in your case? Is it really that GPU demanding or might the slow-down caused by not enough VRAM ? If you have something a unskilled person like me could use and test, I could throw it into a 12GB VRAM card and see what happens...

[PatchWorKs]

Originally Posted by Selur

Without:
a. pytorch support
b. rewriting of the exitistn plugins
-> no

Well, pytorch support SEEMS on the go:
https://github.com/pytorch/pytorch/issues/47702

Originally Posted by Selur

I can run it for single pics fine, but I get like 0.005fps for sd->hd on my system, which simply is too slow for me to be usable.

Of course (that's why a colab can help), but can you please post some visual results ?

Thx !

[Selur]

here are a few examples:

used:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DeinterlaceFilter/TIVTC/libtivtc.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/d2vsource.dll")
# source: 'E:\clips\VTS_02_1-Sample-Beginning.demuxed.m2v'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine
# Loading E:\clips\VTS_02_1-Sample-Beginning.demuxed.m2v using D2VSource
clip = core.d2v.Source(input="E:/Temp/m2v_5d36292e1f7f53fd6e26be51d50bbf8c_853323747.d2v")
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 29.97
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip)
clip = core.tivtc.TDecimate(clip=clip)# new fps: 23.976
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
# DEBUG: vsTIVTC changed scanorder to: progressive
# cropping the video to 704x480
clip = core.std.CropRel(clip=clip, left=8, right=8, top=0, bottom=0)
from vsswinir import SwinIR
# adjusting color space from YUV420P8 to RGBS for VsSwinIR
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# resizing using SwinIR
clip = SwinIR(clip=clip, task="real_sr_large", scale=4, tile_x=352, tile_y=240, tile_pad=16, device_type="cuda", device_index=0) # 2816x1920
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1474, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

some more using RealSR_large:




Cu Selur

[PatchWorKs]

Very nice results (especially on the faces), even if - of course - not yet optimal for everything...

...btw I hope to see a SwinIR version optimized for videos too.

[Selur]

not yet optimal for everything...

Espeically the last example shows some real issues

...btw I hope to see a SwinIR version optimized for videos too.

first a way faster version would be needed

[PatchWorKs]

Originally Posted by Selur

first a way faster version would be needed

Already asked, of course: https://github.com/JingyunLiang/SwinIR/issues/47

Note: I've also just "fed" @HolyWu with this awesome collection, let's see if other interesting "VS-ports" will come out...

[ReinerSchweinlin]

Originally Posted by PatchWorKs

Very nice results (especially on the faces), even if - of course - not yet optimal for everything...

...btw I hope to see a SwinIR version optimized for videos too.

Is a video version planed?

for upscaling, algos like esrgan (single image) are not very suitable for real-life content. Too much flickering. So unless SwinIR doesn´t get some extensions for multi-frame usage / flow detection /whatever, one will always get flickering / stutters / inkonsistent movement...

[aegisofrime]

Nvidia just open sourced Nvidia image scaling... Would this be a candidate for the next filter?

[Selur]

Has anyone tested https://github.com/HolyWu/vs-hinet ?

Here are a few screen shots: (not sure what to make of them and for what content this is really useful)

Mode: Deblur GoPro

Mode: Deblur REDS

Mode: denoise

Mode: derain

[PatchWorKs]

According to your tests on that frame, the highest fidelity seems to be achieved by derain model, btw here are some questions:

1. how fast is it ?
2. how does it performs (in terms of both speed and fidelity) compared to xClean ?
3. its description claims that it perform "restoring" (aka denoising ?) function, but since is BasicSR-based does it upscale too ?

Last but not least (even if OT): did you tried RIFE ?
https://github.com/HolyWu/vs-rife

[Selur]

speed: ~2-3fps for sd content, so not that slow
xClean: no clue about xClean, haven't played around with it too many options for my taste (+ would need to add znedi3 and nnedi3cl support to it)
upscale: at least the current interface offers no upscaling and the method does not upscale
rife: yes, I like it (with sceneChange added). Waiting for FrameRateConverter to properly support it