Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th August 2019, 08:29   #4841  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 5,962
Quote:
Originally Posted by StainlessS View Post
Links with page numbers are a bit unreliable. People could have customized their "posts per page". Best use the "Link".

__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 26th August 2019, 09:11   #4842  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,261
Thanks, not intentional, fixed.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 26th August 2019, 09:51   #4843  |  Link
Groucho2004
Cantankerous Fossil
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Location: A wretched hive of scum and villainy
Posts: 4,470
Quote:
Originally Posted by VoodooFX View Post
This script is 71 times faster in AVS v2.6 vs AVS+ :
Classic Avisynth caches these 60 frames:
Code:
a = clp.Trim(20,-30)+clp.Trim(2000,-30)
and re-uses that block of memory for all loop and/or splice operations. Setting setmemorymax() to 128 or 64 shows what's happening.

AVS+ seems to read from the source every time in this scenario.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 26th August 2019, 17:18   #4844  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,256
For historical reasons Trim - along with other filters - is declared internally as a NonCachedGenericVideoFilter.
Changing that back to GenericVideoFilter yields significant speed increase.

Such filters are: Trim, FreezeFrame, DeleteFrame, DuplicateFrame, Reverse and Loop
pinterf is offline   Reply With Quote
Old 26th August 2019, 18:06   #4845  |  Link
VoodooFX
Registered User
 
Join Date: Sep 2008
Posts: 81
Quote:
Originally Posted by pinterf View Post
For historical reasons Trim - along with other filters - is declared internally as a NonCachedGenericVideoFilter.
Changing that back to GenericVideoFilter yields significant speed increase.

Such filters are: Trim, FreezeFrame, DeleteFrame, DuplicateFrame, Reverse and Loop
If it is only historical reason can you change them back to cached GenericVideoFilter?
VoodooFX is offline   Reply With Quote
Old 26th August 2019, 18:36   #4846  |  Link
filler56789
SuperVirus
 
filler56789's Avatar
 
Join Date: Jun 2012
Location: Antarctic Japan
Posts: 950
Quote:
Originally Posted by VoodooFX View Post
If it is only historical reason can you change them back to cached GenericVideoFilter?
I second that. "History" should be no excuse for bugs /design_flaws /bad_performance.
filler56789 is offline   Reply With Quote
Old 26th August 2019, 19:09   #4847  |  Link
Groucho2004
Cantankerous Fossil
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Location: A wretched hive of scum and villainy
Posts: 4,470
I'm sure pylorak had a good reason for that commit, if I recall correctly it was related to MT functionality.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 26th August 2019, 19:26   #4848  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Ikea Chair
Posts: 2,035
Quote:
Originally Posted by filler56789 View Post
I second that. "History" should be no excuse for bugs /design_flaws /bad_performance.
The reason for not adding caches is simple: avoid insane amounts of useless cache instances from being created. This is a very reasonable tradeoff since the max depth of sane uncached operations is about 10 or so. Not many millions. In typical cases the reduced cache overhead will provide a speedup. That's right! It goes faster for everybody except you!

What you've done here is implement the video scripting equivalent of an extremely inefficient algorithm. I personally view this like someone implementing bubble sort and then complaining about poor performance. Avisynth is a programming language and nobody can protect you from your own shortcomings. It's similar to how Matlab is really slow unless you figure out a way to express things as matrix operations. You have to work with the programming language, NOT AGAINST IT.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 26th August 2019, 20:30   #4849  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 1,293
maybe as compromise and neutral Solution, trim can has cache parameter (false by default) if FrameCache() or RequestLinear() can't fix the slowdown
__________________
My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 26th August 2019, 20:45   #4850  |  Link
VoodooFX
Registered User
 
Join Date: Sep 2008
Posts: 81
I understand that such code doesn't make sense, until you know that it is the only way to get desirable result from external plugin.
VoodooFX is offline   Reply With Quote
Old 26th August 2019, 21:08   #4851  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,084
Quote:
Originally Posted by Groucho2004 View Post
I'm sure pylorak had a good reason for that commit, if I recall correctly it was related to MT functionality.
Early 2014 was when the MT functionality was being implemented and the caching behavior was refactored in tandem with it. That's almost certainly the reason, even if just tangentially.

I mean, the other thing to point out is that the tests are being run single-threaded, and some of that syntax is wonky (probably minuscule in any possible performance detriment, but whatever). Actually using MT shows how artificial this test is in the first place, because it scales with # of cores in virtually 1:1 fashion.

Script (with the cruft trimmed out and clarified):
Code:
clp = FFVideoSource("test.avi").ConvertToRGB24()
a = clp.Trim(20,-30)+clp.Trim(2000,-30)
a1 = a+a+a+a+a+a+a+a+a+a
a1+a1+a1+a1+a1+a1+a1+a1+a1+a1+a1+a1+a1+a1+a1+a1+a1+a1+a1+a1
Prefetch(4)
Without Prefetch:
Code:
>avsmeter test.avs

AVSMeter 2.9.6 (x86) - Copyright (c) 2012-2019, Groucho2004
AviSynth+ 0.1 (r2883, MT, i386) (0.1.0.0)

Number of frames:                12000
Length (hh:mm:ss.ms):     00:08:20.500
Frame width:                       196
Frame height:                       88
Framerate:                      23.976 (24000/1001)
Colorspace:                      RGB24

Frames processed:               12000 (0 - 11999)
FPS (min | max | average):      381.3 | 2361 | 1977
Process memory usage (max):     44 MiB
Thread count:                   16
CPU usage (average):            35.7%

Time (elapsed):                 00:00:06.070
With Prefetch(4):
Code:
>avsmeter test.avs

AVSMeter 2.9.6 (x86) - Copyright (c) 2012-2019, Groucho2004
AviSynth+ 0.1 (r2883, MT, i386) (0.1.0.0)

Number of frames:                12000
Length (hh:mm:ss.ms):     00:08:20.500
Frame width:                       196
Frame height:                       88
Framerate:                      23.976 (24000/1001)
Colorspace:                      RGB24

Frames processed:               12000 (0 - 11999)
FPS (min | max | average):      2164 | 12104 | 10044
Process memory usage (max):     51 MiB
Thread count:                   20
CPU usage (average):            57.5%

Time (elapsed):                 00:00:01.195
64-bit results, for good measure:
Without Prefetch:
Code:
>avsmeter64 test.avs

AVSMeter 2.9.6 (x64) - Copyright (c) 2012-2019, Groucho2004
AviSynth+ 0.1 (r2883, MT, x86_64) (0.1.0.0)

Number of frames:                12000
Length (hh:mm:ss.ms):     00:08:20.500
Frame width:                       196
Frame height:                       88
Framerate:                      23.976 (24000/1001)
Colorspace:                      RGB24

Frames processed:               12000 (0 - 11999)
FPS (min | max | average):      948.1 | 2770 | 2303
Process memory usage (max):     25 MiB
Thread count:                   16
CPU usage (average):            39.1%

Time (elapsed):                 00:00:05.211
With Prefetch(4):
Code:
>avsmeter64 test.avs

AVSMeter 2.9.6 (x64) - Copyright (c) 2012-2019, Groucho2004
AviSynth+ 0.1 (r2883, MT, x86_64) (0.1.0.0)

Number of frames:                12000
Length (hh:mm:ss.ms):     00:08:20.500
Frame width:                       196
Frame height:                       88
Framerate:                      23.976 (24000/1001)
Colorspace:                      RGB24

Frames processed:               12000 (0 - 11999)
FPS (min | max | average):      1767 | 13659 | 11223
Process memory usage (max):     32 MiB
Thread count:                   20
CPU usage (average):            55.7%

Time (elapsed):                 00:00:01.069
With Prefetch(5):
Code:
>avsmeter64 test.avs

AVSMeter 2.9.6 (x64) - Copyright (c) 2012-2019, Groucho2004
AviSynth+ 0.1 (r2883, MT, x86_64) (0.1.0.0)

Number of frames:                12000
Length (hh:mm:ss.ms):     00:08:20.500
Frame width:                       196
Frame height:                       88
Framerate:                      23.976 (24000/1001)
Colorspace:                      RGB24

Frames processed:               12000 (0 - 11999)
FPS (min | max | average):      1692 | 139899 | 23581
Process memory usage (max):     33 MiB
Thread count:                   21
CPU usage (average):            34.1%

Time (elapsed):                 00:00:00.509
If I up the Prefetch value on the 32-bit test to 5 (one higher than the number of cores in this J3455), avsmeter then reports that it's too short to measure. 64-bit hits the same barrier at Prefetch(6). Of course, that's for 12000 frames at a frame size of 196x88; who knows how much more you can abuse the frame threading beyond n cores with frames at a normal size and runtime length.

Last edited by qyot27; 26th August 2019 at 21:13.
qyot27 is offline   Reply With Quote
Old 26th August 2019, 22:26   #4852  |  Link
filler56789
SuperVirus
 
filler56789's Avatar
 
Join Date: Jun 2012
Location: Antarctic Japan
Posts: 950
Quote:
Originally Posted by Myrsloik View Post
The reason for not adding caches is simple: avoid insane amounts of useless cache instances from being created. This is a very reasonable tradeoff since the max depth of sane uncached operations is about 10 or so. Not many millions. In typical cases the reduced cache overhead will provide a speedup. That's right! It goes faster for everybody except you!

What you've done here is implement the video scripting equivalent of an extremely inefficient algorithm. I personally view this like someone implementing bubble sort and then complaining about poor performance. Avisynth is a programming language and nobody can protect you from your own shortcomings. It's similar to how Matlab is really slow unless you figure out a way to express things as matrix operations. You have to work with the programming language, NOT AGAINST IT.
1) Thanks for the useful information. Then I stand corrected, sir.

2) And then I don't know why pinterf wrote ""historical" reasons" instead of giving a straight-to-the-point answer like yours. Perhaps that was a misplaced joke or/and his "sense of humor" is broken......

P.S.:

3) I am not VoodooFX alright

Last edited by filler56789; 26th August 2019 at 22:40.
filler56789 is offline   Reply With Quote
Old 27th August 2019, 00:35   #4853  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 1,840
Quote:
Originally Posted by Myrsloik View Post
What you've done here is implement the video scripting equivalent of an extremely inefficient algorithm. I personally view this like someone implementing bubble sort and then complaining about poor performance. Avisynth is a programming language and nobody can protect you from your own shortcomings. It's similar to how Matlab is really slow unless you figure out a way to express things as matrix operations. You have to work with the programming language, NOT AGAINST IT.
So - and I'm probably being dim here, or missing some MT thing because I've never really dug into that - how do you implement a loop of a small clip section, in Avisynth+, so you can get the same performance as seen in 2.6?
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline   Reply With Quote
Old 27th August 2019, 19:20   #4854  |  Link
VoodooFX
Registered User
 
Join Date: Sep 2008
Posts: 81
Quote:
Originally Posted by qyot27 View Post
Actually using MT shows how artificial this test is in the first place, because it scales with # of cores in virtually 1:1 fashion.
Loading time of the real script (link to - pseudo script, the part where the real script is parsing loops when loading):
Code:
2s   - AVS Standard 2.6
61s  - AVS+
304s - AVS+ with Prefetch(4)
Trims, loops replaced with SelectRangeEvery (~3600 frames from clip):
Code:
41s  - AVS Standard 2.6
46s  - AVS+
247s - AVS+ with Prefetch(4)

Last edited by VoodooFX; 29th August 2019 at 14:44.
VoodooFX is offline   Reply With Quote
Old 27th August 2019, 20:36   #4855  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,256
Quote:
Originally Posted by qyot27 View Post
Early 2014 was when the MT functionality was being implemented and the caching behavior was refactored in tandem with it. That's almost certainly the reason, even if just tangentially.
Setting the non-caching behaviour for zero-op filters in general seemed to be a good idea at that time in 2014, but as this use case shows the idea does not work for Trim and the probably for the other functions mentioned above. In the past few years I have already put back some deleted invoking of InternalCache to Avisynth+ since some real world (and not synthetic) scripts were slow in specific scenarios. Non-caching Trim probably has no penalty only when it appears alone and right after a source filter (? this has to be tested).
pinterf is offline   Reply With Quote
Old 27th August 2019, 23:01   #4856  |  Link
Zetti
Registered User
 
Join Date: Dec 2015
Location: Denmark
Posts: 212
Is there a ETA for a new stable build?
Zetti is offline   Reply With Quote
Old 29th August 2019, 00:09   #4857  |  Link
ajp_anton
Registered User
 
ajp_anton's Avatar
 
Join Date: Aug 2006
Location: Stockholm/Helsinki
Posts: 782
Bug?

I don't know if this is supposed to be "expected behavior", but at least I didn't expect anything like this.

When a resizer is fed cropping coordinates from outside of the picure, what exactly is supposed to happen? I expected it to simply repeat the last line of the image, but that doesn't seem to be the case.

Code:
function blackborder(clip c,int pix)
{
  return c.crop(0,pix,0,-pix).addborders(0,pix,0,pix)
}

blankclip(width=100, height=100, length=20, pixel_type="y8", color=$ffffff)

animate(0,19,"blackborder", 0, 19)

lanczos4resize(width,height, 0, -height/2, width, height*2)
histogram
This creates a white image, but with a black border the same size as the framenumber. It then "zooms out", but keeps the overall dimensions the same, by taking a larger area around the frame and resizing it to the original size.

On frame 0, the expected behavior happens, where it just repeats the white border indefinitely and the zoomed-out image is also 100% white. On frame 1 and onwards, I was expecting it to treat the outside world as being black, however that's not what happens.

Depending on how thick the black border is and what resizer is used, the outside color oscillates around gray/black, mimicking the wave-like nature and the sampling area of the resizer. For complete blackness, a 6 pixel border is needed for spline36, 2 for bilinear, etc.

Last edited by ajp_anton; 29th August 2019 at 00:11.
ajp_anton is offline   Reply With Quote
Old 29th August 2019, 07:45   #4858  |  Link
ajp_anton
Registered User
 
ajp_anton's Avatar
 
Join Date: Aug 2006
Location: Stockholm/Helsinki
Posts: 782
ConvertBits(8,dither=1)
doesn't seem to work for 8-bit sources.

Wiki says:
"Dithering is allowed only for 10-16bit (not 32bit float) sources."
You can dither from 16bits to 16bits so why not from 8bits to 8bits? I'm assuming those will just pass through the video untouched anyway. Life gets easier when I can just slap on a convert to 8bits at the end without worrying about what the input bitdepth is.
ajp_anton is offline   Reply With Quote
Old 29th August 2019, 08:25   #4859  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 5,962
Dithering is a technique to spread quality loss while reducing the precision. Converting from 8 bits per color component to 8 bits, there is no reduction, thus no loss to be spread.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 29th August 2019, 08:48   #4860  |  Link
ajp_anton
Registered User
 
ajp_anton's Avatar
 
Join Date: Aug 2006
Location: Stockholm/Helsinki
Posts: 782
Quote:
Originally Posted by LigH View Post
Dithering is a technique to spread quality loss while reducing the precision. Converting from 8 bits per color component to 8 bits, there is no reduction, thus no loss to be spread.
Yes, I know. So why is 16bits to 16bits allowed?
edit: I'm not saying 16b->16b actually does anything. I'm just confused over the fact that it's allowed, while 8b->8b isn't. I'm advocating for allowing also 8b->8b for when you can just add it to the end of a script or a function, without adding a check to see if it's allowed... (I feel like I'm repeating myself here, everything is already in the original post).

Last edited by ajp_anton; 29th August 2019 at 08:50.
ajp_anton is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:12.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.