vs-basicvsr

[poisondeathray]

Yet another contribution from HolyWu!

Thanks

https://github.com/HolyWu/vs-basicvsr

This one is true temporal SR for video, not single images

Also has tiling option for low GPU memory vs. task

Nice!

[Zarxrax]

What is this difference between vs-basicvsr, vs-realesrgan, and VSGAN?
As far as I can see, it looks like vs-basicvsr & vs-realesrgan have built-in models that you can use, wheras VSGAN lets you just use whatever model you want.
Is there something else that I am missing?

[poisondeathray]

Originally Posted by Zarxrax

What is this difference between vs-basicvsr, vs-realesrgan, and VSGAN?
As far as I can see, it looks like vs-basicvsr & vs-realesrgan have built-in models that you can use, wheras VSGAN lets you just use whatever model you want.
Is there something else that I am missing?

Yes,

other differences - vsgan implementation of ESRGAN does not have tiling (image is split up with padding and stitched up back togethe) - this helps when your GPU has low memory compared to image size; some other ESRGAN implementations have tiling. Quality wise you cannot tell the difference with large enough padding even zoomed in on still images - although it is detectable with amplified differences.

basicvsr is temporal - in general there is less aliasing and temporal flicker. It's meant for video and works on more than 1 frame. The others are meant for still images (except tecogan, but that does not have a VPY implementation). Single image based SR has less temporal consistency. Unfortunately I'm finding the 3 models provided for basicvsr generally only work for fairly clean sources. Just like with other GAN's you generally have to pre and post process to "massage" the input for a suitable model. It would be nice if you had a specific model for everything and thousands of different situations - I don't see it happening

But users can train their own models for basicvsr and realesrgan - the training code is published - but there isn't a public repository for them - only the published paper models are readily available. In theory you should be able to use other pre-trained models too, with minor adjustment to the script

[aegisofrime]

HolyWu is on a roll, holy molly.

Hoping to see VS implementations of AMD FSR or Intel XeSS next

[kedautinh12]

Originally Posted by aegisofrime

HolyWu is on a roll, holy molly.

Hoping to see VS implementations of AMD FSR or Intel XeSS next

I think Nvidia dlss is better but sad it's close source

[aegisofrime]

Originally Posted by kedautinh12

I think Nvidia dlss is better but sad it's close source

Yeah, that was the reason I didn't mention DLSS, unless nvidia decides to open source it due to competitive pressure.

[Quadratic]

Originally Posted by aegisofrime

HolyWu is on a roll, holy molly.

Hoping to see VS implementations of AMD FSR or Intel XeSS next

https://github.com/GPUOpen-Effects/F...fsr/ffx_fsr1.h
AMD FSR (EASU) is quite interesting, I would imagine one would probably be able to approximate the results with existing tools - but an actual implementation would be nice.

[Julek]

DLSS even if open would not be easy to port, the models are trained specifically for each game, there is no generic model.
But if you have an RTX card, you can at least use DLISR, via the Akarin plugin.
https://github.com/AkarinVS/vapoursynth-plugin#dlisr

[aegisofrime]

Originally Posted by Julek

DLSS even if open would not be easy to port, the models are trained specifically for each game, there is no generic model.
But if you have an RTX card, you can at least use DLISR, via the Akarin plugin.
https://github.com/AkarinVS/vapoursynth-plugin#dlisr

OMG, I have been looking for a DLSS VS plugin for ages now, but I guess I either didn't bother Googling for it or my Google skills suck. Thanks for linking this! I wonder how many other plugins lie under my nose...

[edcrfv94]

Originally Posted by Julek

DLSS even if open would not be easy to port, the models are trained specifically for each game, there is no generic model.
But if you have an RTX card, you can at least use DLISR, via the Akarin plugin.
https://github.com/AkarinVS/vapoursynth-plugin#dlisr

DLSS 2.0 is generic model, but still need motion vector.
This means it cannot be used for video.

[aegisofrime]

Originally Posted by edcrfv94

DLSS 2.0 is generic model, but still need motion vector.
This means it cannot be used for video.

According to the plugin description, it is using NGX though which is indeed designed for video applications.

[kedautinh12]

Originally Posted by aegisofrime

According to the plugin description, it is using NGX though which is indeed designed for video applications.

I think DLSS don't same DLISR

[Selur]

vs-basicvsr
Yet another contribution from HolyWu!

Nice, and there goes the weekend,..

[Selur]

Btw. has anyone an example where BasicVSR show some impressive results when starting from a 480p or 576p source?
Normal resizing using i.e. Lanczos and adding some contrast sharpening seems to produce more impressive results.
Only thing that I found is that it does do well is upscaling QCIF content, and I wonder whether that is it's goal.

Cu Selur

[poisondeathray]

Originally Posted by Selur

Btw. has anyone an example where BasicVSR show some impressive results when starting from a 480p or 576p source?
Normal resizing using i.e. Lanczos and adding some contrast sharpening seems to produce more impressive results.
Only thing that I found is that it does do well is upscaling QCIF content, and I wonder whether that is it's goal.

BasicVSR works ok in simular situations to where the ESRGAN base model works ok - ie. fairly clean, downsampled sources - really an oversampled SD source... Which is NOT what people typically have - it's usually some noisy, soft DVD source, or SD crappy home video that people want to upscale. But those clean datasets are what most publically available algorithms were trained on: Clean HR ground truth source, downscaled by 4. Usually with bicubic. BasicVSR used selections from the Reds4 and Vimeo90k datasets. qcif "suzie" is really an oversampled source. If you take some ok quality UHD, and downscale it, that will work too. What's needed is training of specific models that take account for noise, degredation, compression artifacts like mpeg2 artifacts, etc... stuff like bad YT compression and degredation after the downscaling step. There are specific datasets that include noise - but they are synthetic noise - and they weren't used for these default models .

So in those situations where the model is applicable - BasicVSR is going to produce more detailed images than , say lanczos3 + any form of sharpen, but less detailed results than default ESRGAN or ESRGAN+ models on single image examination. On single image, ESRGAN+ is going to look more "wow" in applicable source situations. The main difference is BasicVSR is more consistent temporally, when you look at actual video, or more than 1 image. It resembles tecogan in that regard, but produces better results detail wise than the default tecogan model (which isn't the same as the tecogan published paper model). For all single image SR approaches you often need to use some type of temporal smoothing to make it decent for video purposes (and that smoothing reduces the fine details too) , otherwise you are prone to flickering aliasing artifacts, objects changing shapes (as single image prediction is slightly different on different frames) .

So the question is can you use something like ESRGAN, add some QTGMC in progressive mode - to get similar temporally consistent results to BasicVSR, but hopefully keep more fine details? The answer is generally no. The artifacts are too "baked" in - that's the strength of basicvsr

[poisondeathray]

Some apng comparisons. They should animate and loop in most browsers

1:1 non-AR corrected crops, unfiltered. 720x480 => 2880x1920.

Note the temporal flickering artifacts (aliasing flicker, and prediction flicker).

You can see the earrings look like random lightning in the ESRGAN+ results.



No amount of sharpening will transform lanczos3 into as detailed as BasicVSR. No amount of temporal filtering will transform lanczos3, or ESRGAN+ results into as temporally consistent results as BasicVSR




The license plate, you can make out the numbers/letters with lanczos, but it's much more clear in the BasicVSR. It doesn't take a stretch of the imagination that slightly smaller or a more distant plate might be illegible when processed with lanczos , but partially readable by BasicVSR



ESRGAN and psnr model mess up letters/numbers/text (many GAN, NN models do), I won't include in the animated temporal demo, but here is a still image compare animated demo. ESRGAN default does good on BG foliage textures, but messes up text. None of the GAN scalers have "fuzzy" lines or bad aliasing - that's what they excel at. Early gen NNEDI/1/2/3 is NN (neural net) based, but signficantly worse in terms of aliasing artifacts for upscaling 4x than modern GAN scalers





But these are cleanly downsample source examples, much like the training set. ie. They don't have MPEG2 macroblocks or or grain plate added on top like you might with some film DVD

Some trained degraded jpeg ESRGAN models work ok for single image jpeg interpolation - I would like to see more BasicVSR models that utilize some typical degraded aspects of common video

[PatchWorKs]

Originally Posted by Selur

Normal resizing using i.e. Lanczos and adding some contrast sharpening seems to produce more impressive results.

HolyWu has just ported - after my request - SwinIR to VS, can someone make some "real world" test with it ?

https://github.com/HolyWu/vs-swinir

Originally Posted by Selur

man this is too slow on my machine to be useful for normal usage on my gpu (Geforce GTX 1070ti)

For individual frames (= pics) you can test it here, but a video-oriented colab notebook like this would be great (I don't own a discrete GPU at all) !


EDIT
Out of curiosity: do you think the new Apple chips (M1 Pro / Max) could speed up operations?

[ChaosKing]

@Selur I think the link is broken.

[Selur]

Sorry, fixed now.

[kedautinh12]

The link work for me but almost German