vs-basicvsr

[PatchWorKs]

Originally Posted by poisondeathray

PP-MSVSR featured in PaddleGAN looks like the next improvement over basicvsr++ . I'm trying to get it to work, but no luck so far with modifying config files and paths

...to me could be VERY interesting too:

OverNet: Lightweight multi-scale super-resolution with overscaling network

https://arxiv.org/abs/2008.02382

Unfortunally their GH seems empty:
https://github.com/pbehjatii/OverNet

[ReinerSchweinlin]

thanx for mentioning it,m looking good

there is hope, GITHUB says:

This repository is an official PyTorch implementation of the paper "OverNet: Lightweight multi-scale super-resolution with overscaling network" from WACV 2021. The code will be released soon.

[poisondeathray]

I requested VRT: A Video Restoration Transformer

https://github.com/JingyunLiang/VRT

https://github.com/HolyWu/vs-basicvsrpp/issues/21

[PatchWorKs]

Here's another kind of enhancer that claims to make HDR from SDR:
https://github.com/nothinglo/Deep-Photo-Enhancer



Results on video:
https://www.youtube.com/watch?v=d7OXb2sqoec

Pytorch implementation:
https://github.com/mtics/deep-photo-enhancer

[Selur]

HolyWu posted an update for vs-dpir. I'm not sure what provider uses what onxruntime.
Okay, got that cleared up, see: https://github.com/HolyWu/vs-dpir/issues/20

CUDA:
encoded 429 frames, 8.83 fps, 1117.06 kb/s
DIRECTML:
encoded 429 frames, 6.03 fps, 1117.74 kb/s
CPU:
encoded 429 frames, 0.49 fps, 1116.08 kb/s

(can't test CUDA TensorRT since I only got an old Geforce GTX 1070ti)
-> seems that onnxruntime-gpu is the way to go for CUDA cards.

[cubicibo]

Originally Posted by PatchWorKs

Here's another kind of enhancer that claims to make HDR from SDR:
https://github.com/nothinglo/Deep-Photo-Enhancer

[Image]

I wouldn't say this is an improvement. Look at the halo around the "PUMA" on the shirt.

[Selur]

Did anyone get vs-dpir installed when using Vapoursynth R58-RC1? (https://github.com/HolyWu/vs-dpir/issues/22)
vs-basicvsrpp also doesn't work since mmcv_full is meant for Python 3.9 which isn't supported in Vapoursynth R58 (only 3.8 and 3.10) (https://github.com/HolyWu/vs-basicvsrpp/issues/23)
Similar for vs-rife and vs-realesrgan.

Cu Selur

[Selur]

Thanks to HolyWu I got vs-basicvsrpp working using:

python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp

Cu Selur

[ReinerSchweinlin]

Originally Posted by Selur

Thanks to HolyWu I got vs-basicvsrpp working using:

python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp

Cu Selur

Nice Thanx to you both

[PatchWorKs]

Originally Posted by Selur

Thanks to HolyWu I got vs-basicvsrpp working using:

python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp

Cu Selur

Holy & Slur contributions & testing tandem rockz !

[PatchWorKs]

How about (new ?) VRT - A Video Restoration Transformer ?

https://github.com/JingyunLiang/VRT

Tests would be great.

[Selur]

Maybe vs-mlrt (https://github.com/AmusementClub/vs-mlrt) will add support for it, based onhttps://docs.microsoft.com/en-us/windows/ai/windows-ml/tutorials/pytorch-convert-model conversion from pth to onnx models might be possible.

[WolframRhodium]

Sadly, ther non-supported operations (deformable convolution and others) in VRT prevent the conversion to the standard onnx format.

[Selur]

argh,.. good to know, thanks for looking into it.

[~ VEGETA ~]

is there any good use of this filter on high quality anime encode releases?

I mean if sharpness and detail preservation are wanted, then descaling is the way to go if can be done. Plus some high quality denoisers and careful masked debanding can yield best results.

what can this filter offer?

[Selur]

Assuming you are referring to BasicVSR++, it's meant for restoration so using it on a 'high quality' sounds like wanting to give a new car to a restaurator,...
As a side note: There are tons of Anime trained models (https://upscale.wiki/wiki/Model_Database) for VSGAN which might be more suited, detail preservation totally depends on the model used.
Using ml stuff on high resolution content is really slow with current hardware, so if high quality also means high source resolution be prepared to wait.

Cu Selur

[poisondeathray]

I've modified __init__.py in the vsbasicvsrpp folder to include the NTIRE 4x VSR model if anyone is interested

https://www.mediafire.com/file/k0bba...+VSR).zip/file

https://github.com/open-mmlab/mmedit...icvsr_plusplus

Extract __init__.py from the zip archive ,and download the model and place in same vsbasicvsrpp directory
https://download.openmmlab.com/mmedi...1-1ff35292.pth

        model: Model to use.
            0 = REDS
            1 = Vimeo-90K (BI)
            2 = Vimeo-90K (BD)
            3 = NTIRE 2021 Video Super-Resolution
            4 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 1
            5 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 2
            6 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 3

I inserted it as model "3" because models 0 to 2 are 4x models, and NTIRE 2021 Video Super-Resolution is a 4x model too; and that bumps the NTIRE 1x models down

The model uses mid_channels 128 and num_blocks 25 (instead of 64 and 7), as written in the pth name, and in the py file, so I modified those entries and the other relevant ones
https://github.com/open-mmlab/mmedit...k_ntire_vsr.py

It seems to work ok, and results are similar to REDS model, maybe slightly better or on some sources. The py says it was trained on REDS data set with spynet_20210409-c6c1bd09.pth pretrain

[Selur]

Nice.Thanks!
(added a request in the vs-basicvsrpp bugtacker to add this)

[Selur]

btw. HolyWu just release vs-dpir-ncnn version

python -m pip install -U vsdpir_ncnn
python -m pip install --upgrade https://github.com/HolyWu/ncnn/releases/download/1.0.20220910/ncnn-1.0.20220910-cp310-cp310-win_amd64.whl
python -m vsdpir_ncnn

usage:

from vsdpir_ncnn import dpir

ret = dpir(clip)

options:

def dpir(
    clip: vs.VideoNode,
    strength: float | vs.VideoNode | None = None,
    task: str = 'denoise',
    tile_w: int = 0,
    tile_h: int = 0,
    tile_pad: int = 8,
    gpu_id: int | None = None,
    fp16: bool = True,
) -> vs.VideoNode:
    """
    DPIR: Deep Plug-and-Play Image Restoration
    Parameters:
        clip: Clip to process. Only RGB and GRAY formats with float sample type of 32 bit depth are supported.
        strength: Strength for deblocking/denoising. Defaults to 50.0 for 'deblock', 5.0 for 'denoise'.
            Also accepts a GRAY8/GRAYS clip for varying strength.
        task: Task to perform. Must be 'deblock' or 'denoise'.
        tile_w, tile_h: Tile width and height, respectively. As too large images result in the out of GPU memory issue,
            so this tile option will first crop input images into tiles, and then process each of them.
            Finally, they will be merged into one image. 0 denotes for do not use tile.
        tile_pad: The pad size for each tile, to remove border artifacts.
        gpu_id: The GPU ID.
        fp16: Enable FP16 mode.
    """

------------------------------


If someone compares this to https://github.com/HolyWu/vs-dpir let us know whether there are some speed differences between the two.
Thanks!

On my Geforce GTX 1070ti on 640x352 content I get:

vs-dpir:

clip = DPIR(clip=clip, strength=5.000, task="denoise", provider=1, device_id=0, dual=True)

Output 429 frames in 96.04 seconds (4.47 fps) (without dual=True, I get 4.29fps)

clip = DPIR(clip=clip, strength=5.000, task="denoise", provider=1, device_id=0)

Output 429 frames in 96.04 seconds (4.47 fps)

vs-dpir-ncnn:

clip = dpir(clip=clip, strength=5.000, task="denoise", gpu_id=0)

Output 429 frames in 117.23 seconds (3.66 fps)

So for me it's better to stick with normal vs-dpir, but I wonder whether this is true for folks with other cards.

Cu Selur

[Selur]

update to vs-dpir-ncnn:

v2.0.0
Repository: HolyWu/vs-dpir-ncnn · Tag: v2.0.0 · Commit: 0938c18 · Released by: HolyWu
Turn dpir function into a class so as to separate model initialization and inference. It's more memory friendly when you run the same task more than once (such as on different clips or with different strengths) since the same model will be initialized only once.

Cu Selur