Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

 
 
Thread Tools Search this Thread Display Modes
Prev Previous Post   Next Post Next
Old 28th September 2019, 22:19   #1  |  Link
TomArrow
Registered User
 
Join Date: Dec 2017
Posts: 90
Someone able to write a CUDA plugin for ESRGAN models? (Mind-blowing upscales!)

There's a fantastic upscale tool out there called ESRGAN. See repository:

https://github.com/xinntao/ESRGAN/

There's two parts to the process: Training and applying (or testing, as the author calls it). For the training you use a dataset that closely resembles your final dataset, but you need high resolution versions for that one too too. The upscaling by default is x4.

For training there's a separate repository with training codes: https://github.com/xinntao/BasicSR

Then when applying, you take the trained model and apply it to low resolution frames you don't have any high resolution counteprart for.

The only way to do this currently is with this repository by extracting all frames as PNGs. This is okay for training, but for applying it's an extreme time-waster. It would be fantastic if there was a way to do this within a CUDA-capable AVS filter, for example like this:

clip.ESRGAN("mymodel.pth")

I'm a complete C++ beginner and would struggle to even write a normal plugin and the mere idea of writing a CUDA plugin is like a wild dream for me.

But the advantage of this upscaling over the normal AVISynth upscaling filters is so incredibly big, I figure maybe someone here with the necessary skills might feel inclined to do this. I've seen it smoothing compression artifacts just through training it against a high resolution source without compression artifacts, I've seen it clean up lines, I've seen it automatically generate organic looking noise that resembles the reference it's trained against, even fix colors, all through the power of training with a good dataset.

If any of you heard of Gigapixel, this is exactly like that, but much better, because you can train it for your particular content.

To demonstrate this, here's an example from the movie Dark Knight where I trained a DVD against the HD version of the movie: http://www.framecompare.com/image-co...rison/DKZYLNNX

As you can see, Lanczos3 doesn't stand a chance.

I figure someone with the necessary skills could just look at the python code and adapt it for C++.

It's not really necessary to do this for the training part, but for the actual execution part it would be an immense time- and space-saver. Where normally you would need terabytes of free space to process a longer video (due to the default x4 magnification), with a plugin you could downscale to something reasonable (like 1080p) on-the-fly and then encode straight to x264 or similar. It would save the time of extracting the source frames, time of encoding pngs, and then also the time of encoding result pngs and having to read them again.

In the end, people could train their models, then share them with each other to use on each other's content, various models tailored to various codecs, types of compression artifacts, types of content, resolutions etc. One model is around 100MB big, so very very easy to share, even though the training can take weeks.

Of course only the more tech-savvy people would ever get into training, but lots of normal folks could easily apply the models to their own projects, which would be amazing.

So far, the best way for normal users to realistically use AI upscaling is to extract frames as PNGs, load them into Gigapixel (which they have to buy) and then live with the fact that it's not really trained for their kind of content and will have unbecoming artifacts all over the place. I think this could really revolutionize video restoration.

Thoughts?

Last edited by TomArrow; 28th September 2019 at 22:29.
TomArrow is offline   Reply With Quote
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:55.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.