Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 25th August 2021, 03:38   #1  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,332
vs-basicvsr

Yet another contribution from HolyWu!

Thanks

https://github.com/HolyWu/vs-basicvsr

This one is true temporal SR for video, not single images

Also has tiling option for low GPU memory vs. task

Nice!
poisondeathray is offline   Reply With Quote
Old 25th August 2021, 22:12   #2  |  Link
Zarxrax
Registered User
 
Join Date: Dec 2001
Posts: 1,219
What is this difference between vs-basicvsr, vs-realesrgan, and VSGAN?
As far as I can see, it looks like vs-basicvsr & vs-realesrgan have built-in models that you can use, wheras VSGAN lets you just use whatever model you want.
Is there something else that I am missing?
Zarxrax is offline   Reply With Quote
Old 25th August 2021, 23:38   #3  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,332
Quote:
Originally Posted by Zarxrax View Post
What is this difference between vs-basicvsr, vs-realesrgan, and VSGAN?
As far as I can see, it looks like vs-basicvsr & vs-realesrgan have built-in models that you can use, wheras VSGAN lets you just use whatever model you want.
Is there something else that I am missing?
Yes,

other differences - vsgan implementation of ESRGAN does not have tiling (image is split up with padding and stitched up back togethe) - this helps when your GPU has low memory compared to image size; some other ESRGAN implementations have tiling. Quality wise you cannot tell the difference with large enough padding even zoomed in on still images - although it is detectable with amplified differences.

basicvsr is temporal - in general there is less aliasing and temporal flicker. It's meant for video and works on more than 1 frame. The others are meant for still images (except tecogan, but that does not have a VPY implementation). Single image based SR has less temporal consistency. Unfortunately I'm finding the 3 models provided for basicvsr generally only work for fairly clean sources. Just like with other GAN's you generally have to pre and post process to "massage" the input for a suitable model. It would be nice if you had a specific model for everything and thousands of different situations - I don't see it happening

But users can train their own models for basicvsr and realesrgan - the training code is published - but there isn't a public repository for them - only the published paper models are readily available. In theory you should be able to use other pre-trained models too, with minor adjustment to the script

Last edited by poisondeathray; 25th August 2021 at 23:41.
poisondeathray is offline   Reply With Quote
Old 26th August 2021, 02:17   #4  |  Link
aegisofrime
Registered User
 
Join Date: Apr 2009
Posts: 478
HolyWu is on a roll, holy molly.

Hoping to see VS implementations of AMD FSR or Intel XeSS next
aegisofrime is offline   Reply With Quote
Old 26th August 2021, 02:25   #5  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 2,127
Quote:
Originally Posted by aegisofrime View Post
HolyWu is on a roll, holy molly.

Hoping to see VS implementations of AMD FSR or Intel XeSS next
I think Nvidia dlss is better but sad it's close source
kedautinh12 is offline   Reply With Quote
Old 26th August 2021, 02:46   #6  |  Link
aegisofrime
Registered User
 
Join Date: Apr 2009
Posts: 478
Quote:
Originally Posted by kedautinh12 View Post
I think Nvidia dlss is better but sad it's close source
Yeah, that was the reason I didn't mention DLSS, unless nvidia decides to open source it due to competitive pressure.
aegisofrime is offline   Reply With Quote
Old 26th August 2021, 12:04   #7  |  Link
Quadratic
Registered User
 
Join Date: Jul 2021
Posts: 26
Quote:
Originally Posted by aegisofrime View Post
HolyWu is on a roll, holy molly.

Hoping to see VS implementations of AMD FSR or Intel XeSS next
https://github.com/GPUOpen-Effects/F...fsr/ffx_fsr1.h
AMD FSR (EASU) is quite interesting, I would imagine one would probably be able to approximate the results with existing tools - but an actual implementation would be nice.
Quadratic is offline   Reply With Quote
Old 26th August 2021, 15:11   #8  |  Link
Julek
Registered User
 
Julek's Avatar
 
Join Date: Dec 2020
Posts: 83
DLSS even if open would not be easy to port, the models are trained specifically for each game, there is no generic model.
But if you have an RTX card, you can at least use DLISR, via the Akarin plugin.
https://github.com/AkarinVS/vapoursynth-plugin#dlisr
Julek is offline   Reply With Quote
Old 26th August 2021, 15:17   #9  |  Link
aegisofrime
Registered User
 
Join Date: Apr 2009
Posts: 478
Quote:
Originally Posted by Julek View Post
DLSS even if open would not be easy to port, the models are trained specifically for each game, there is no generic model.
But if you have an RTX card, you can at least use DLISR, via the Akarin plugin.
https://github.com/AkarinVS/vapoursynth-plugin#dlisr
OMG, I have been looking for a DLSS VS plugin for ages now, but I guess I either didn't bother Googling for it or my Google skills suck. Thanks for linking this! I wonder how many other plugins lie under my nose...
aegisofrime is offline   Reply With Quote
Old 26th August 2021, 20:25   #10  |  Link
edcrfv94
Registered User
 
Join Date: Apr 2015
Posts: 84
Quote:
Originally Posted by Julek View Post
DLSS even if open would not be easy to port, the models are trained specifically for each game, there is no generic model.
But if you have an RTX card, you can at least use DLISR, via the Akarin plugin.
https://github.com/AkarinVS/vapoursynth-plugin#dlisr
DLSS 2.0 is generic model, but still need motion vector.
This means it cannot be used for video.
edcrfv94 is offline   Reply With Quote
Old 27th August 2021, 02:10   #11  |  Link
aegisofrime
Registered User
 
Join Date: Apr 2009
Posts: 478
Quote:
Originally Posted by edcrfv94 View Post
DLSS 2.0 is generic model, but still need motion vector.
This means it cannot be used for video.
According to the plugin description, it is using NGX though which is indeed designed for video applications.
aegisofrime is offline   Reply With Quote
Old 27th August 2021, 02:48   #12  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 2,127
Quote:
Originally Posted by aegisofrime View Post
According to the plugin description, it is using NGX though which is indeed designed for video applications.
I think DLSS don't same DLISR
kedautinh12 is offline   Reply With Quote
Old 26th August 2021, 17:56   #13  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,231
Quote:
vs-basicvsr
Yet another contribution from HolyWu!
Nice, and there goes the weekend,..
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 27th August 2021, 21:06   #14  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,231
Btw. has anyone an example where BasicVSR show some impressive results when starting from a 480p or 576p source?
Normal resizing using i.e. Lanczos and adding some contrast sharpening seems to produce more impressive results.
Only thing that I found is that it does do well is upscaling QCIF content, and I wonder whether that is it's goal.

Cu Selur
__________________
Hybrid here in the forum, homepage

Last edited by Selur; 27th August 2021 at 21:59.
Selur is offline   Reply With Quote
Old 28th August 2021, 14:10   #15  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,332
Quote:
Originally Posted by Selur View Post
Btw. has anyone an example where BasicVSR show some impressive results when starting from a 480p or 576p source?
Normal resizing using i.e. Lanczos and adding some contrast sharpening seems to produce more impressive results.
Only thing that I found is that it does do well is upscaling QCIF content, and I wonder whether that is it's goal.

BasicVSR works ok in simular situations to where the ESRGAN base model works ok - ie. fairly clean, downsampled sources - really an oversampled SD source... Which is NOT what people typically have - it's usually some noisy, soft DVD source, or SD crappy home video that people want to upscale. But those clean datasets are what most publically available algorithms were trained on: Clean HR ground truth source, downscaled by 4. Usually with bicubic. BasicVSR used selections from the Reds4 and Vimeo90k datasets. qcif "suzie" is really an oversampled source. If you take some ok quality UHD, and downscale it, that will work too. What's needed is training of specific models that take account for noise, degredation, compression artifacts like mpeg2 artifacts, etc... stuff like bad YT compression and degredation after the downscaling step. There are specific datasets that include noise - but they are synthetic noise - and they weren't used for these default models .

So in those situations where the model is applicable - BasicVSR is going to produce more detailed images than , say lanczos3 + any form of sharpen, but less detailed results than default ESRGAN or ESRGAN+ models on single image examination. On single image, ESRGAN+ is going to look more "wow" in applicable source situations. The main difference is BasicVSR is more consistent temporally, when you look at actual video, or more than 1 image. It resembles tecogan in that regard, but produces better results detail wise than the default tecogan model (which isn't the same as the tecogan published paper model). For all single image SR approaches you often need to use some type of temporal smoothing to make it decent for video purposes (and that smoothing reduces the fine details too) , otherwise you are prone to flickering aliasing artifacts, objects changing shapes (as single image prediction is slightly different on different frames) .

So the question is can you use something like ESRGAN, add some QTGMC in progressive mode - to get similar temporally consistent results to BasicVSR, but hopefully keep more fine details? The answer is generally no. The artifacts are too "baked" in - that's the strength of basicvsr
poisondeathray is offline   Reply With Quote
Old 28th August 2021, 17:52   #16  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,332
Some apng comparisons. They should animate and loop in most browsers

1:1 non-AR corrected crops, unfiltered. 720x480 => 2880x1920.

Note the temporal flickering artifacts (aliasing flicker, and prediction flicker).

You can see the earrings look like random lightning in the ESRGAN+ results.



No amount of sharpening will transform lanczos3 into as detailed as BasicVSR. No amount of temporal filtering will transform lanczos3, or ESRGAN+ results into as temporally consistent results as BasicVSR




The license plate, you can make out the numbers/letters with lanczos, but it's much more clear in the BasicVSR. It doesn't take a stretch of the imagination that slightly smaller or a more distant plate might be illegible when processed with lanczos , but partially readable by BasicVSR



ESRGAN and psnr model mess up letters/numbers/text (many GAN, NN models do), I won't include in the animated temporal demo, but here is a still image compare animated demo. ESRGAN default does good on BG foliage textures, but messes up text. None of the GAN scalers have "fuzzy" lines or bad aliasing - that's what they excel at. Early gen NNEDI/1/2/3 is NN (neural net) based, but signficantly worse in terms of aliasing artifacts for upscaling 4x than modern GAN scalers





But these are cleanly downsample source examples, much like the training set. ie. They don't have MPEG2 macroblocks or or grain plate added on top like you might with some film DVD

Some trained degraded jpeg ESRGAN models work ok for single image jpeg interpolation - I would like to see more BasicVSR models that utilize some typical degraded aspects of common video
poisondeathray is offline   Reply With Quote
Old 6th November 2021, 08:11   #17  |  Link
PatchWorKs
Registered User
 
PatchWorKs's Avatar
 
Join Date: Aug 2002
Location: Italy
Posts: 299
Quote:
Originally Posted by Selur View Post
Normal resizing using i.e. Lanczos and adding some contrast sharpening seems to produce more impressive results.
HolyWu has just ported - after my request - SwinIR to VS, can someone make some "real world" test with it ?

https://github.com/HolyWu/vs-swinir



Quote:
Originally Posted by Selur View Post
man this is too slow on my machine to be useful for normal usage on my gpu (Geforce GTX 1070ti)
For individual frames (= pics) you can test it here, but a video-oriented colab notebook like this would be great (I don't own a discrete GPU at all) !


EDIT
Out of curiosity: do you think the new Apple chips (M1 Pro / Max) could speed up operations?

Last edited by PatchWorKs; 6th November 2021 at 08:50.
PatchWorKs is offline   Reply With Quote
Old 27th August 2021, 21:20   #18  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,788
@Selur I think the link is broken.
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database
ChaosKing is offline   Reply With Quote
Old 27th August 2021, 22:00   #19  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,231
Sorry, fixed now.
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 28th August 2021, 00:38   #20  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 2,127
The link work for me but almost German
kedautinh12 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 03:53.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.