Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Announcements and Chat > General Discussion

Reply
 
Thread Tools Search this Thread Display Modes
Old 14th May 2023, 17:02   #1  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Deep Denoise Temporal

Trying to find nn denoisers in the temporal domain.
Here is one that seems interesting:
https://github.com/clausmichele/ViDeNN
Has anyone here tried it or has any other suggestions of temporal nn denoisers?

Edit: more links
https://github.com/ZhouYiiFeng/TDMS-Net

Last edited by anton_foy; 14th May 2023 at 17:15.
anton_foy is offline   Reply With Quote
Old 15th May 2023, 19:58   #2  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by anton_foy View Post
Trying to find nn denoisers in the temporal domain.
Here is one that seems interesting:
https://github.com/clausmichele/ViDeNN
Has anyone here tried it or has any other suggestions of temporal nn denoisers?
Never tried this one, but it runs ok in Windows

Ok results so far on the test sets , but nothing spectacular

No strength adjustment

Sometimes there is a loss of a frame , I think based on the way it works if there is a scene change, such as in the table tennis set. You have to be careful if you're using PSNR or metrics to align the original vs. results

Has CPU mode fallback , but no tiling options

Older smaller neural net, relatively faster than larger NN. eg
VRT deniosing model about 5-6x larger and more than that many times slower
https://github.com/JingyunLiang/VRT




Quote:
Edit: more links
https://github.com/ZhouYiiFeng/TDMS-Net
Haven't tried this, but you have to install some baidu application for downloading the model. Not going to do that
poisondeathray is offline   Reply With Quote
Old 16th May 2023, 07:49   #3  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Thanks for that!
Quote:
Ok results so far on the test sets , but nothing spectacular

No strength adjustment

Sometimes there is a loss of a frame , I think based on the way it works if there is a scene change, such as in the table tennis set. You have to be careful if you're using PSNR or metrics to align the original vs. results

Has CPU mode fallback , but no tiling options
Ok and no adjustment for temporal radius either I guess. Speed and quality compared to mdegrainN is not close then for either ViDeNN or VRT would you say?
anton_foy is offline   Reply With Quote
Old 16th May 2023, 10:49   #4  |  Link
PatchWorKs
Registered User
 
PatchWorKs's Avatar
 
Join Date: Aug 2002
Location: Italy
Posts: 303
We strongly suggest to check the User-friendly, Fast, Self-Supervised Image Denoising for All project:
https://github.com/royerlab/aydin/
__________________
HYbrid Multimedia Production Suite project @ GitHub
PatchWorKs is offline   Reply With Quote
Old 16th May 2023, 13:43   #5  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by anton_foy View Post
Thanks for that!


Ok and no adjustment for temporal radius either I guess. Speed and quality compared to mdegrainN is not close then for either ViDeNN or VRT would you say?

No adjustments for ViDeNN (other than model choice) , it's slower than mdegrain derivatives.

VRT has adjustments for temporal frame : width: height tiling, otherwise you'd need massive amounts of GPU memory, and strength 0-50 . Speed is very, very, very, very slow. It's got to be the one of slowest NN processing algorithms (not just for denoising)

The resulting quality depends on the noise type for the pretrained models. Both tend to do better on gaussian type noise, not on stuff like large grain, because that's what they were trained on. If you look at the table tennis example in ViDeNN, mdegrain derivatives typically won't do very well on that type of noise, but ViDeNN will do ok, and scunet will do great (even though it's spatial, there will be no temporal inconsistencies in that particular case)
poisondeathray is offline   Reply With Quote
Old 16th May 2023, 16:55   #6  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
@PatchWorks
Thanks looks interesting, it says also that it is spatiotemporal. Must look into, but it only seems to take still sequencies.

@Poisondeathray
Interesting that scunet doesn't leave those temporal inconsistencies yet probably because high ISO and low light footage has too little information especially in the darker parts while the tennis clip has added noise that does not distort in that way.

Edit:
Using Davinci Resolve's NR alot at work and while it smears (motionblurs) alot it has a great ability at low settings to remove lighter temporal fluctuations and blotches from bad compression.

Last edited by anton_foy; 16th May 2023 at 17:00.
anton_foy is offline   Reply With Quote
Old 16th May 2023, 17:08   #7  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by anton_foy View Post
@Poisondeathray
Interesting that scunet doesn't leave those temporal inconsistencies yet probably because high ISO and low light footage has too little information especially in the darker parts while the tennis clip has added noise that does not distort in that way.
Scunet can leave temporal inconsistencies - in general. All spatial only algorithms can.

But on that specific table tennis sequence it does not, and you don't even need to add other smoothing afterwards. ViDNN OTOH does leave some temporal fluctuations on that sequence even though it's temporal. Scunet retains more detail and is cleaner, more temporally stable than ViDNN on that sequence . I mention the table tennis sequence because that's the highlight presented for ViDNN

Last edited by poisondeathray; 16th May 2023 at 17:12.
poisondeathray is offline   Reply With Quote
Old 16th May 2023, 17:39   #8  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Quote:
Originally Posted by poisondeathray View Post
Scunet can leave temporal inconsistencies - in general. All spatial only algorithms can.

But on that specific table tennis sequence it does not, and you don't even need to add other smoothing afterwards. ViDNN OTOH does leave some temporal fluctuations on that sequence even though it's temporal. Scunet retains more detail and is cleaner, more temporally stable than ViDNN on that sequence . I mention the table tennis sequence because that's the highlight presented for ViDNN
Scunet is also in the slower category you would say?
anton_foy is offline   Reply With Quote
Old 16th May 2023, 17:59   #9  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by anton_foy View Post
Scunet is also in the slower category you would say?
Yes - All the NN algorithms are several orders slower than mvtools2 derivatives or typical avs denoisers. The temporal ones like VRT can be exponentially slower unless you have a quadro with 48GB GPU memory

Scunet is only good for specific types of noise with the pretrained models, but it falls in the "remarkable" category. Very useful tool, sometimes as a prefilter for some avs/vpy filters . It makes you wonder how it would behave with custom training sets
poisondeathray is offline   Reply With Quote
Old 17th July 2023, 18:28   #10  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
I found this clip where they are using what they call "Overlap Loss" for temporal consistency:

https://youtu.be/LA747HTukTQ
anton_foy is offline   Reply With Quote
Old 17th July 2023, 18:45   #11  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by anton_foy View Post
I found this clip where they are using what they call "Overlap Loss" for temporal consistency:

https://youtu.be/LA747HTukTQ
Looks interesting .

Only paper link , no test code published that I could find .

Might have to wait a few months, sometimes people eventually publish an implementation of a paper on github
poisondeathray is offline   Reply With Quote
Old 18th July 2023, 08:18   #12  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Quote:
Originally Posted by poisondeathray View Post
Looks interesting .

Only paper link , no test code published that I could find .

Might have to wait a few months, sometimes people eventually publish an implementation of a paper on github
Yes I could only find this

and test code

but it is only for temporal consistencies and I have no idea how useful it is for real footage. Edit: the kinetics videos used there are broken.

Edit: and this and this
but it is for superres.

Last edited by anton_foy; 18th July 2023 at 08:33.
anton_foy is offline   Reply With Quote
Old 18th July 2023, 13:42   #13  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
I know little to nothing of A.I. generative techniques, but thought that this link to a Tom'sHARDWARE article,
posted via CodeProject might be of interest to some, might even give hints to potential problems in other A.I. disciplines.
Generative AI Goes 'MAD' When Trained on AI-Created Data Over Five Times:- https://www.tomshardware.com/news/ge...ver-five-times
Quote:
Generative AI goes "MAD" after five training iterations on artificial outputs.
Quote:
Model Autophagy Disorder isn't confirmed to affect all AI models, but the researchers did verify it against autoencoders, Gaussian mixture models, and large language models.

It just so happens that all of these types of models that can "go MAD" have been widespread and operating for a while now: autoencoders can handle things such as popularity prediction (in things such as a social media app's algorithm), image compression, image denoising, and image generation; and gaussian mixture models are used for density estimation, clustering, and image segmentation purposes, which makes them particularly useful for statistical and data sciences.
Quote:
This research provides a way to peer into the black box of AI development. And it shreds any hope that we'd found an endless fountain of data from making a hamster wheel out of certain AI models: feeding it data, and then feeding its own data back into it, in order to generate more data that's then fed back again.

This could be an issue for currently-existing models and applications of these models: if a model that's achieved commercial use has, in fact, been trained on its own outputs, then that model has likely regressed toward its mean (remember it takes around five cycles of input-output for that to manifest). And if that model has regressed towards its mean, then it's been biased in some way, shape, or form, as it doesn't consider the data that would naturally be in the minority. Algorithmic bigotry, if you will.
[ClickMe]


(a sort of 'A.I. ringing')
Perhaps pertinent for eg, Super Resolution generative techniques.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 18th July 2023 at 15:15.
StainlessS is offline   Reply With Quote
Old 18th July 2023, 23:40   #14  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Quote:
Originally Posted by StainlessS View Post
I know little to nothing of A.I. generative techniques, but thought that this link to a Tom'sHARDWARE article,
posted via CodeProject might be of interest to some, might even give hints to potential problems in other A.I. disciplines.
Generative AI Goes 'MAD' When Trained on AI-Created Data Over Five Times:- https://www.tomshardware.com/news/ge...ver-five-times





[ClickMe]


(a sort of 'A.I. ringing')
Perhaps pertinent for eg, Super Resolution generative techniques.
Yes I have seen this over at the Vapoursynth section with some super res esrgan stuff. But my main interest is the temporal consitency with denoising. I got Chat GPT to write a simple avs+ script that works ok actually stabilizing temporally and spatial denoise. Just for fun that is besides the point of this thread

Last edited by anton_foy; 19th July 2023 at 06:56.
anton_foy is offline   Reply With Quote
Old 29th July 2023, 08:14   #15  |  Link
PatchWorKs
Registered User
 
PatchWorKs's Avatar
 
Join Date: Aug 2002
Location: Italy
Posts: 303
Sorry for the delay in reply, we're quite busy.

Quote:
Originally Posted by anton_foy View Post
@PatchWorks
Thanks looks interesting, it says also that it is spatiotemporal. Must look into, but it only seems to take still sequencies.
Here's an interesting reply about by aydin's author @ my GitHub "issue":
Quote:
Originally Posted by Ahmet Can Solak
I want to clarify that Aydin works on spatio-temporal data(TZYX) when all four dimensions are available and can do TYX as well. Aydin doesn't read video file formats directly though, if you can save it as a file format that is supported then you can denoise movies too. With biological live imaging datasets, we use Aydin for denoising movies on daily basis.
Hope that helps.
__________________
HYbrid Multimedia Production Suite project @ GitHub
PatchWorKs is offline   Reply With Quote
Old 29th July 2023, 11:20   #16  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Quote:
Originally Posted by PatchWorKs View Post
Sorry for the delay in reply, we're quite busy.



Here's an interesting reply about by aydin's author @ my GitHub "issue":

Hope that helps.
Thanks I will try on my mac I have problems with python on my PC.
anton_foy is offline   Reply With Quote
Old 31st August 2023, 09:41   #17  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Ok this is a shader but it seems interesting and I wonder if it can be used to reduce noise instead of just AA. Only previous frame for motion vectors though but it looks quite stable and the nvidia paper linked in the dropboxlink mentions reduced noise and reduced ghosting. Maybe possible to tweak or replace the AA part with spatial denoising (since AA is spatial in it self)?

Link: https://www.shadertoy.com/view/MscSD7

Last edited by anton_foy; 31st August 2023 at 10:08.
anton_foy is offline   Reply With Quote
Old 28th September 2023, 11:28   #18  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
It looks current cameras have enough effective internal denoisers (and/or pro shooting conditions provide enough light) so development of good denoisers for processing after shooting close to stopped.
So developers of 'neural networks' make very little progress in this way. In previous decades when we had a big market for consumer video cameras of medium and poor (cheap) quality - there was higher demand on post-shooting denoise (but PCs were slow). So it looks the era of best development of denoise solutions is spreaded in some years of ending of consumer video cameras and beginning of not very slow PCs at home.

Currently even if some business will invest in development of denoise solution - there is no market for it to return the investment (like was in the old times of Neat Video products for consumer market). And pro denoise solutions for video cameras are closed in the production of video cameras for both pro and smartphone video cameras. The RAW shooting 'video modes' of consumer photo cameras may expect good lighting conditions and not shipped with good RAW denoise post-processing software. And typical large-sensor consumer photo-cameras do not have enough processing hardware resources onboard to perform good temporal denoise.
DTL is offline   Reply With Quote
Old 30th October 2023, 16:39   #19  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Another interesting thing perhaps:
https://sreyas-mohan.github.io/udvd/
https://github.com/sreyas-mohan/udvd

Up to 5 frames temporal radius.
anton_foy is offline   Reply With Quote
Old 30th October 2023, 21:17   #20  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by anton_foy View Post
Another interesting thing perhaps:
https://sreyas-mohan.github.io/udvd/
https://github.com/sreyas-mohan/udvd

Up to 5 frames temporal radius.
There are PSNR rankings for many of these on commonly used test datasets (DAVIS , and set8) on the paperswithcode website

udvd gets about 2-3db lower than VRT/RVRT and ReVid . PSNR isn't the best metric, and certainly doesn't measure any temporal component (unlike VMAF), but that delta is large enough that any metric will show similar trend. ReVid is the top on every dataset there, but there is no public code or model available yet .

https://paperswithcode.com/task/video-denoising

Some of the older papers have other comparisons like Neat Video for the dB included in the results chart, but it's not clear how Neat Video was used, or if it was used properly. Neat does surprisingly poorly, I suspect not used correctly. Might be fun to run current Neat video or some avs/vpy denoisers on one of the known datasets to compare
https://github.com/m-tassano/dvdnet#psnrs-set8-testset

I think currently VRT/RVRT are probably the best "working" ones, with configurable settings, ... but soooooooo slow . And you really need to train your own models for the best results, unless your sources are similar to DAVIS
poisondeathray is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 10:23.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.