Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 28th September 2019, 22:19   #1  |  Link
TomArrow
Registered User
 
Join Date: Dec 2017
Posts: 90
Someone able to write a CUDA plugin for ESRGAN models? (Mind-blowing upscales!)

There's a fantastic upscale tool out there called ESRGAN. See repository:

https://github.com/xinntao/ESRGAN/

There's two parts to the process: Training and applying (or testing, as the author calls it). For the training you use a dataset that closely resembles your final dataset, but you need high resolution versions for that one too too. The upscaling by default is x4.

For training there's a separate repository with training codes: https://github.com/xinntao/BasicSR

Then when applying, you take the trained model and apply it to low resolution frames you don't have any high resolution counteprart for.

The only way to do this currently is with this repository by extracting all frames as PNGs. This is okay for training, but for applying it's an extreme time-waster. It would be fantastic if there was a way to do this within a CUDA-capable AVS filter, for example like this:

clip.ESRGAN("mymodel.pth")

I'm a complete C++ beginner and would struggle to even write a normal plugin and the mere idea of writing a CUDA plugin is like a wild dream for me.

But the advantage of this upscaling over the normal AVISynth upscaling filters is so incredibly big, I figure maybe someone here with the necessary skills might feel inclined to do this. I've seen it smoothing compression artifacts just through training it against a high resolution source without compression artifacts, I've seen it clean up lines, I've seen it automatically generate organic looking noise that resembles the reference it's trained against, even fix colors, all through the power of training with a good dataset.

If any of you heard of Gigapixel, this is exactly like that, but much better, because you can train it for your particular content.

To demonstrate this, here's an example from the movie Dark Knight where I trained a DVD against the HD version of the movie: http://www.framecompare.com/image-co...rison/DKZYLNNX

As you can see, Lanczos3 doesn't stand a chance.

I figure someone with the necessary skills could just look at the python code and adapt it for C++.

It's not really necessary to do this for the training part, but for the actual execution part it would be an immense time- and space-saver. Where normally you would need terabytes of free space to process a longer video (due to the default x4 magnification), with a plugin you could downscale to something reasonable (like 1080p) on-the-fly and then encode straight to x264 or similar. It would save the time of extracting the source frames, time of encoding pngs, and then also the time of encoding result pngs and having to read them again.

In the end, people could train their models, then share them with each other to use on each other's content, various models tailored to various codecs, types of compression artifacts, types of content, resolutions etc. One model is around 100MB big, so very very easy to share, even though the training can take weeks.

Of course only the more tech-savvy people would ever get into training, but lots of normal folks could easily apply the models to their own projects, which would be amazing.

So far, the best way for normal users to realistically use AI upscaling is to extract frames as PNGs, load them into Gigapixel (which they have to buy) and then live with the fact that it's not really trained for their kind of content and will have unbecoming artifacts all over the place. I think this could really revolutionize video restoration.

Thoughts?

Last edited by TomArrow; 28th September 2019 at 22:29.
TomArrow is offline   Reply With Quote
Old 28th September 2019, 22:33   #2  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
why CUDA not OpenCL?
__________________
See My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 28th September 2019, 22:41   #3  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
You can use ESRGAN and other models in vapoursynth.

As a starting point: (there is no easy one click setup solution)
https://github.com/WolframRhodium/mu...sfunc.py#L4329
https://github.com/WolframRhodium/Super-Resolution-Zoo
+ you need to install CUDA and some other python stuff

See also https://forum.doom9.org/showthread.php?t=175721
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database
ChaosKing is offline   Reply With Quote
Old 28th September 2019, 22:59   #4  |  Link
TomArrow
Registered User
 
Join Date: Dec 2017
Posts: 90
Quote:
Originally Posted by real.finder View Post
why CUDA not OpenCL?
No reason really, other than the fact that I assumed it has to be CUDA to be compatible. If OpenCL works just as well, it's ultimately the same thing, no problem.
TomArrow is offline   Reply With Quote
Old 28th September 2019, 23:01   #5  |  Link
TomArrow
Registered User
 
Join Date: Dec 2017
Posts: 90
Quote:
Originally Posted by ChaosKing View Post
You can use ESRGAN and other models in vapoursynth.

As a starting point: (there is no easy one click setup solution)
https://github.com/WolframRhodium/mu...sfunc.py#L4329
https://github.com/WolframRhodium/Super-Resolution-Zoo
+ you need to install CUDA and some other python stuff

See also https://forum.doom9.org/showthread.php?t=175721
Wew, that's a bit much too take in right now, but glad to see someone already made this possible, might definitely be a good reason to finally check out VapourSynth. However, I see that Super-Resolution-Zoo has the models converted to some other format. Is this something that I can do with my own models too or is that something only the repository author knows how to do?
TomArrow is offline   Reply With Quote
Old 28th September 2019, 23:11   #6  |  Link
Cary Knoop
Cary Knoop
 
Cary Knoop's Avatar
 
Join Date: Feb 2017
Location: Newark CA, USA
Posts: 397
Currently, all those deep learning environments are loaded with gotchas, the only flow they have is a flow of tensions!

The first who one who makes a development environment that simply works instead of some hacked-together-it-works-but-it depends assembly deserves a Nobel prize.
Cary Knoop is offline   Reply With Quote
Old 29th September 2019, 00:35   #7  |  Link
WolframRhodium
Registered User
 
Join Date: Jan 2016
Posts: 162
Quote:
Originally Posted by TomArrow View Post
However, I see that Super-Resolution-Zoo has the models converted to some other format. Is this something that I can do with my own models too or is that something only the repository author knows how to do?
Yes you can definitely do it yourself. I'm glad to help you, in case you have any questions.
WolframRhodium is offline   Reply With Quote
Old 29th September 2019, 12:19   #8  |  Link
Natty
Noob
 
Join Date: Mar 2017
Posts: 221
an avisynth plugin based on this would be awesome ��
Natty is offline   Reply With Quote
Old 2nd October 2019, 11:50   #9  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
I found also this https://github.com/AlphaAtlas/Vapour...olution-Helper
Haven't tested it myself, but should help one with the super res setup.
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database
ChaosKing is offline   Reply With Quote
Old 6th November 2019, 00:56   #10  |  Link
AvengerOne
Registered User
 
Join Date: Oct 2019
Posts: 3
Here a script that uses only two images to process input-output for waifu2x, and can easily be modified for any program (gimp, waifu2x, ImageMagick, ESRGAN, python, etc.)
And it has a simple duplicate frame detector.

https://forum.doom9.org/showthread.php?t=177058

Sample:
https://youtu.be/kdbhFVZ4vIM
AvengerOne is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 13:14.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.