Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 18th March 2019, 17:34   #1  |  Link
ABurns
Registered User
 
ABurns's Avatar
 
Join Date: Aug 2017
Posts: 25
Possible to MT TemporalDegrain in AVS+?

I'm in the (seems like years long) process of converting my entire DVD and BD library to x265 for both archiving and for use in my media server, and one of the most common denoisers I use is TemporalDegrain(degrain=1,ov=2,hq=1). Unfortunately, this function doesn't seem to have been updated in a good many years now and it doesn't multi-thread well. Running it on OC 5820k CPU and AVS+ 2772 with Prefetch(6) and feeding it to x264 works fine on SD material from DVD rips, and it encodes in real time at close to 100% CPU usage (using x264 --crf 20 --preset veryslow)

But now that I've moved into my 1080 BD collection, feeding that same script to x265 results in 4.5fps @ ~20% CPU. For comparison, I can comment out the denoising line and encode with the same settings in real time @ ~100% CPU (x265 --crf 23). So something in TDG is bottlenecking it.

I did some searching, and it looks like there was/is a TemporalDegrain_MT() that multithreads it while using a single thread for the prefilter clip, but it seems to be written for the old 2.6 MT() structure and not the new AVS+ style.

Is there a way to effectively MT this function in AVS+? I really like it a lot because it preserves so much fine detail compared to other denoisers, and at a faster speed than comparable denoisers like MCTemoiralDenoise(). In a worst-case I can just encode multiple jobs in parallel and get 4-5fps out of each, but I'd rather have one job running at 12 or 15 fps and 100% CPU if possible.

Anyone have thoughts?
ABurns is offline   Reply With Quote
Old 18th March 2019, 21:33   #2  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 6,080
Quote:
Anyone have thoughts?
Sadly I have no clue how to make TDG better threading wise, but:
Have you tried using gpu=true ? (this way instead of fft3dfilter, fft3dgpu will be used)

Using fft3dfilter:
Code:
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
FFVideoSource("F:\TESTCL~1\files\MPEG-4~1.264\AVC_PC~2.MOV",cachefile="E:\Temp\mov_b94a578b42f01afe5ec9dfd3bdb37a93_853323747_1_0.ffindex")
TemporalDegrain(degrain=1,ov=2,hq=1)
PreFetch(6)
return last
AvsMeter reports:
Code:
Number of frames:                  276
Length (hh:mm:ss.ms):     00:00:11.512
Frame width:                      1920
Frame height:                     1080
Framerate:                      23.976 (24000/1001)
Colorspace:                       i420

Frames processed:               276 (0 - 275)
FPS (min | max | average):      1.053 | 476191 | 4.220
Process memory usage (max):     2975 MiB
Thread count:                   70
CPU usage (average):            24.8%

Time (elapsed):                 00:01:05.396
When using fft3dgpu ('TemporalDegrain(degrain=1,ov=2,hq=1,gpu=true)')
I get:
Code:
Number of frames:                  276
Length (hh:mm:ss.ms):     00:00:11.512
Frame width:                      1920
Frame height:                     1080
Framerate:                      23.976 (24000/1001)
Colorspace:                       i420

Frames processed:               276 (0 - 275)
FPS (min | max | average):      1.189 | 294118 | 11.79
Process memory usage (max):     1174 MiB
Thread count:                   98
CPU usage (average):            54.1%

Time (elapsed):                 00:00:23.414
-> average speed more than doubled, memory consumption is less then half.

using Prefetch(8) on my Ryzen 7 1800X gives me:
Code:
Frames processed:               276 (0 - 275)
FPS (min | max | average):      1.177 | 86207 | 12.65
Process memory usage (max):     1301 MiB
Thread count:                   100
CPU usage (average):            65.8%

Time (elapsed):                 00:00:21.811
-> trying different Prefetch values might also be a good idea.

Cu Selur
__________________
Hybrid here in the forum, homepage
Notice: Since email notifications do not work here any more, it might take me quite some time to notice a reply to a thread,..
Selur is offline   Reply With Quote
Old 18th March 2019, 21:49   #3  |  Link
ABurns
Registered User
 
ABurns's Avatar
 
Join Date: Aug 2017
Posts: 25
I've thought of that in the past and tried gpu=true, but I've never been able to get gpu to work on my system. AVS throws an access violation without any meaningful error message, so I've never been able to really chase down the problem, and tbh I never really spent that much time on it. Maybe it's worth spending a little time trying to get that to work.

Also, Prefetch(6) seems historically to be the sweet spot for most filters on my system (=physical cores), so I haven't really tweaked it much lately, but I'll give that a shot too.
ABurns is offline   Reply With Quote
Old 18th March 2019, 22:18   #4  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,008
Quote:
Originally Posted by ABurns View Post
Unfortunately, this function doesn't seem to have been updated in a good many years now and it doesn't multi-thread well.
There is a V2 of this script and it apparently works fine multi-threaded.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 18th March 2019, 23:21   #5  |  Link
ABurns
Registered User
 
ABurns's Avatar
 
Join Date: Aug 2017
Posts: 25
Quote:
Originally Posted by Groucho2004 View Post
There is a V2 of this script and it apparently works fine multi-threaded.
Dood! I owe you a beer. I swear I spent 2 days googling it and that didn't turn up in any of my searches. Somebody really needs to update the wiki.

I need to play with the knobs on that because apparently they are completely different, but I did get the GPU thing figured out. I was missing some DX9 libs, and now gpu=true is working. That one change alone got me up to 14.5fps and 100%. I got around 10fps with TDG2 using limitFFT=2, which I think does the same thing, but I need to play with it some before I start using it.

Marking [SOLVED]
[edit: apparently I can't]


Last edited by ABurns; 18th March 2019 at 23:34.
ABurns is offline   Reply With Quote
Old 18th March 2019, 23:54   #6  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,008
Quote:
Originally Posted by ABurns View Post
Somebody really needs to update the wiki.
You mean this Wiki?
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 19th March 2019, 00:11   #7  |  Link
ABurns
Registered User
 
ABurns's Avatar
 
Join Date: Aug 2017
Posts: 25
Quote:
Originally Posted by Groucho2004 View Post
You mean this Wiki?
Yeah, that wiki. You're working on losing that beer.
ABurns is offline   Reply With Quote
Old 19th March 2019, 00:21   #8  |  Link
ABurns
Registered User
 
ABurns's Avatar
 
Join Date: Aug 2017
Posts: 25
Seriously, thanks for the assist. I was searching for "avisynth temporaldegrain MT" and "avisynth temporaldegrain speed" and so forth. It was apparently a case of asking the wrong question.
ABurns is offline   Reply With Quote
Old 19th March 2019, 01:23   #9  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,008
Quote:
Originally Posted by ABurns View Post
You're working on losing that beer.
D'oh.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 20:52.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.