Possible to MT TemporalDegrain in AVS+?

[ABurns]

I'm in the (seems like years long) process of converting my entire DVD and BD library to x265 for both archiving and for use in my media server, and one of the most common denoisers I use is TemporalDegrain(degrain=1,ov=2,hq=1). Unfortunately, this function doesn't seem to have been updated in a good many years now and it doesn't multi-thread well. Running it on OC 5820k CPU and AVS+ 2772 with Prefetch(6) and feeding it to x264 works fine on SD material from DVD rips, and it encodes in real time at close to 100% CPU usage (using x264 --crf 20 --preset veryslow)

But now that I've moved into my 1080 BD collection, feeding that same script to x265 results in 4.5fps @ ~20% CPU. For comparison, I can comment out the denoising line and encode with the same settings in real time @ ~100% CPU (x265 --crf 23). So something in TDG is bottlenecking it.

I did some searching, and it looks like there was/is a TemporalDegrain_MT() that multithreads it while using a single thread for the prefilter clip, but it seems to be written for the old 2.6 MT() structure and not the new AVS+ style.

Is there a way to effectively MT this function in AVS+? I really like it a lot because it preserves so much fine detail compared to other denoisers, and at a faster speed than comparable denoisers like MCTemoiralDenoise(). In a worst-case I can just encode multiple jobs in parallel and get 4-5fps out of each, but I'd rather have one job running at 12 or 15 fps and 100% CPU if possible.

Anyone have thoughts?

[Selur]

Anyone have thoughts?

Sadly I have no clue how to make TDG better threading wise, but:
Have you tried using gpu=true ? (this way instead of fft3dfilter, fft3dgpu will be used)

Using fft3dfilter:

SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
FFVideoSource("F:\TESTCL~1\files\MPEG-4~1.264\AVC_PC~2.MOV",cachefile="E:\Temp\mov_b94a578b42f01afe5ec9dfd3bdb37a93_853323747_1_0.ffindex")
TemporalDegrain(degrain=1,ov=2,hq=1)
PreFetch(6)
return last

AvsMeter reports:

Number of frames:                  276
Length (hh:mm:ss.ms):     00:00:11.512
Frame width:                      1920
Frame height:                     1080
Framerate:                      23.976 (24000/1001)
Colorspace:                       i420

Frames processed:               276 (0 - 275)
FPS (min | max | average):      1.053 | 476191 | 4.220
Process memory usage (max):     2975 MiB
Thread count:                   70
CPU usage (average):            24.8%

Time (elapsed):                 00:01:05.396

When using fft3dgpu ('TemporalDegrain(degrain=1,ov=2,hq=1,gpu=true)')
I get:

Number of frames:                  276
Length (hh:mm:ss.ms):     00:00:11.512
Frame width:                      1920
Frame height:                     1080
Framerate:                      23.976 (24000/1001)
Colorspace:                       i420

Frames processed:               276 (0 - 275)
FPS (min | max | average):      1.189 | 294118 | 11.79
Process memory usage (max):     1174 MiB
Thread count:                   98
CPU usage (average):            54.1%

Time (elapsed):                 00:00:23.414

-> average speed more than doubled, memory consumption is less then half.

using Prefetch(8) on my Ryzen 7 1800X gives me:

Frames processed:               276 (0 - 275)
FPS (min | max | average):      1.177 | 86207 | 12.65
Process memory usage (max):     1301 MiB
Thread count:                   100
CPU usage (average):            65.8%

Time (elapsed):                 00:00:21.811

-> trying different Prefetch values might also be a good idea.

Cu Selur

[ABurns]

I've thought of that in the past and tried gpu=true, but I've never been able to get gpu to work on my system. AVS throws an access violation without any meaningful error message, so I've never been able to really chase down the problem, and tbh I never really spent that much time on it. Maybe it's worth spending a little time trying to get that to work.

Also, Prefetch(6) seems historically to be the sweet spot for most filters on my system (=physical cores), so I haven't really tweaked it much lately, but I'll give that a shot too.

[Groucho2004]

Originally Posted by ABurns

Unfortunately, this function doesn't seem to have been updated in a good many years now and it doesn't multi-thread well.

There is a V2 of this script and it apparently works fine multi-threaded.

[ABurns]

Originally Posted by Groucho2004

There is a V2 of this script and it apparently works fine multi-threaded.

Dood! I owe you a beer. I swear I spent 2 days googling it and that didn't turn up in any of my searches. Somebody really needs to update the wiki.

I need to play with the knobs on that because apparently they are completely different, but I did get the GPU thing figured out. I was missing some DX9 libs, and now gpu=true is working. That one change alone got me up to 14.5fps and 100%. I got around 10fps with TDG2 using limitFFT=2, which I think does the same thing, but I need to play with it some before I start using it.

Marking [SOLVED]
[edit: apparently I can't]

[Groucho2004]

Originally Posted by ABurns

Somebody really needs to update the wiki.

You mean this Wiki?

[ABurns]

Originally Posted by Groucho2004

You mean this Wiki?

Yeah, that wiki. You're working on losing that beer.

[ABurns]

Seriously, thanks for the assist. I was searching for "avisynth temporaldegrain MT" and "avisynth temporaldegrain speed" and so forth. It was apparently a case of asking the wrong question.

[Groucho2004]

Originally Posted by ABurns

You're working on losing that beer.

D'oh.