Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 13th April 2018, 13:13   #3041  |  Link
HolyWu
Registered User
 
HolyWu's Avatar
 
Join Date: Aug 2006
Location: Taiwan
Posts: 486
Well, probably because most users don't even know that they have to change a group policy security setting first in order to make large page work.

Anyway, here is the usual benchmark.

PHP Code:
core.std.BlankClip(width=3840height=2160length=500format=vs.YUV420P10)
core.resize.Bilinear(cformat=vs.RGBSmatrix_in_s='2020ncl'transfer_in_s='709'primaries_in_s='2020',
                         
matrix_s='rgb'transfer_s='linear'primaries_s='2020'dither_type='none')
core.resize.Bilinear(cformat=vs.YUV420P8matrix_in_s='rgb'transfer_in_s='linear'primaries_in_s='709',
                         
matrix_s='709'transfer_s='709'primaries_s='709'dither_type='ordered')
c.set_output() 
Code:
# VS_NO_LARGE_PAGES not set
>vspipe test.vpy .
Windows VirtualAlloc bug detected: page still mapped
Output 500 frames in 40.59 seconds (12.32 fps)

# VS_NO_LARGE_PAGES set
>vspipe test.vpy .
Output 500 frames in 21.43 seconds (23.33 fps)
I don't know whether it's because I only have 8 GB RAM and it's too few for large page?
HolyWu is offline   Reply With Quote
Old 13th April 2018, 13:56   #3042  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 5,598
The only other application I remember having support for large pages is 7-zip.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 15th April 2018, 06:25   #3043  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 5,627
So enabling large page support is a bad thing atm.
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 15th April 2018, 19:35   #3044  |  Link
VS_Fan
Registered User
 
Join Date: Jan 2016
Posts: 76
Shouldn't Linux, macOS and Windows 10 v18.03 users be safe testing with Large Memory Pages? It would be nice to have any benchmarks from them.

Igor Pavlov (developer of 7-zip) posted his findings and thoughts some 5 days ago
VS_Fan is offline   Reply With Quote
Old 15th April 2018, 19:36   #3045  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Ikea Chair
Posts: 1,858
Quote:
Originally Posted by VS_Fan View Post
Shouldn't Linux, macOS and Windows 10 v18.03 users be safe testing with Large Memory Pages? It would be nice to have any benchmarks from them.

Igor Pavlov (developer of 7-zip) posted his findings and thoughts some 5 days ago
It's safe on all windows 10 versions. There's a workaround for the bug. Obviously D9 didn't bother to actually try it but instead went off and made stuff up...
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 16th April 2018, 08:51   #3046  |  Link
VS_Fan
Registered User
 
Join Date: Jan 2016
Posts: 76
Iím on laptop with an intel core i5 4gen, 8GB RAM and windows 10 home edition (v18.03 = 10.0.16299.371), so no group policy editor for me. So I tried running vspipe in a command prompt with administrator rights to get LP working.

This are the results for the same vpy script provided by HolyWu: (vspipe.exe test.vpy .)

Code:
Vapoursynth R43: (No LP)
Output 500 frames in 37.62 seconds (13.29 fps)

Vapoursynth R44-test1, VS_NO_LARGE_PAGES set:
Output 500 frames in 40.75 seconds (12.27 fps)

Vapoursynth R44-test1, VS_NO_LARGE_PAGES not set:
Output 500 frames in 37.51 seconds (13.33 fps)

Vapoursynth R44-test1, VS_NO_LARGE_PAGES not set: (on an administrator command prompt)
Windows VirtualAlloc bug detected: page still mapped
Output 500 frames in 39.67 seconds (12.60 fps)
The time variations could be related to the antivirus scanning my external USB disk in the background
VS_Fan is offline   Reply With Quote
Old 16th April 2018, 09:21   #3047  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Ikea Chair
Posts: 1,858
Quote:
Originally Posted by VS_Fan View Post
Iím on laptop with an intel core i5 4gen, 8GB RAM and windows 10 home edition (v18.03 = 10.0.16299.371), so no group policy editor for me. So I tried running vspipe in a command prompt with administrator rights to get LP working.

This are the results for the same vpy script provided by HolyWu: (vspipe.exe test.vpy .)

Code:
Vapoursynth R43: (No LP)
Output 500 frames in 37.62 seconds (13.29 fps)

Vapoursynth R44-test1, VS_NO_LARGE_PAGES set:
Output 500 frames in 40.75 seconds (12.27 fps)

Vapoursynth R44-test1, VS_NO_LARGE_PAGES not set:
Output 500 frames in 37.51 seconds (13.33 fps)

Vapoursynth R44-test1, VS_NO_LARGE_PAGES not set: (on an administrator command prompt)
Windows VirtualAlloc bug detected: page still mapped
Output 500 frames in 39.67 seconds (12.60 fps)
The time variations could be related to the antivirus scanning my external USB disk in the background
I think your background tasks have a bigger effect than large pages at its best.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 16th April 2018, 23:37   #3048  |  Link
Stephen R. Savage
Registered User
 
Stephen R. Savage's Avatar
 
Join Date: Nov 2009
Posts: 337
Quote:
Originally Posted by HolyWu View Post
Well, probably because most users don't even know that they have to change a group policy security setting first in order to make large page work.

Anyway, here is the usual benchmark.

PHP Code:
core.std.BlankClip(width=3840height=2160length=500format=vs.YUV420P10)
core.resize.Bilinear(cformat=vs.RGBSmatrix_in_s='2020ncl'transfer_in_s='709'primaries_in_s='2020',
                         
matrix_s='rgb'transfer_s='linear'primaries_s='2020'dither_type='none')
core.resize.Bilinear(cformat=vs.YUV420P8matrix_in_s='rgb'transfer_in_s='linear'primaries_in_s='709',
                         
matrix_s='709'transfer_s='709'primaries_s='709'dither_type='ordered')
c.set_output() 
Code:
# VS_NO_LARGE_PAGES not set
>vspipe test.vpy .
Windows VirtualAlloc bug detected: page still mapped
Output 500 frames in 40.59 seconds (12.32 fps)

# VS_NO_LARGE_PAGES set
>vspipe test.vpy .
Output 500 frames in 21.43 seconds (23.33 fps)
I don't know whether it's because I only have 8 GB RAM and it's too few for large page?
You need to increase the VS memory limit (core.max_cache_size) to prevent thrashing. I found that setting it to 16384 (16 GB) was effective. The issue is that VS only allows 10% of the memory limit to be used for allocating memory (the other 90% goes to filter caches).
Stephen R. Savage is offline   Reply With Quote
Old 17th April 2018, 03:23   #3049  |  Link
HolyWu
Registered User
 
HolyWu's Avatar
 
Join Date: Aug 2006
Location: Taiwan
Posts: 486
Quote:
Originally Posted by Stephen R. Savage View Post
You need to increase the VS memory limit (core.max_cache_size) to prevent thrashing. I found that setting it to 16384 (16 GB) was effective. The issue is that VS only allows 10% of the memory limit to be used for allocating memory (the other 90% goes to filter caches).
Good point. The speed does get better when I increase max_cache_size, but it's still slower than not using large pages.

Code:
=== VS_NO_LARGE_PAGES not set ===

# max_cache_size = 4096
Windows VirtualAlloc bug detected: page still mapped
Output 500 frames in 39.42 seconds (12.68 fps)

# max_cache_size = 8192
Windows VirtualAlloc bug detected: page still mapped
Output 500 frames in 25.11 seconds (19.91 fps)

# max_cache_size = 16384
Windows VirtualAlloc bug detected: page still mapped
Output 500 frames in 24.31 seconds (20.57 fps)


=== VS_NO_LARGE_PAGES set ===

# max_cache_size = 4096
Output 500 frames in 22.34 seconds (22.38 fps)

# max_cache_size = 8192
Output 500 frames in 21.38 seconds (23.39 fps)

# max_cache_size = 16384
Output 500 frames in 21.00 seconds (23.81 fps)
HolyWu is offline   Reply With Quote
Old 17th April 2018, 04:47   #3050  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 986
I find the wording of "VS_NO_LARGE_PAGES set/not set" confusing. I have to rethink a few times every time I read it because of the double negative.
lansing is offline   Reply With Quote
Old 17th April 2018, 04:52   #3051  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,380
Quote:
Originally Posted by VS_Fan View Post
I’m on laptop with an intel core i5 4gen, 8GB RAM and windows 10 home edition (v18.03 = 10.0.16299.371), so no group policy editor for me. So I tried running vspipe in a command prompt with administrator rights to get LP working.

This are the results for the same vpy script provided by HolyWu: (vspipe.exe test.vpy .)

Code:
Vapoursynth R43: (No LP)
Output 500 frames in 37.62 seconds (13.29 fps)

Vapoursynth R44-test1, VS_NO_LARGE_PAGES set:
Output 500 frames in 40.75 seconds (12.27 fps)

Vapoursynth R44-test1, VS_NO_LARGE_PAGES not set:
Output 500 frames in 37.51 seconds (13.33 fps)

Vapoursynth R44-test1, VS_NO_LARGE_PAGES not set: (on an administrator command prompt)
Windows VirtualAlloc bug detected: page still mapped
Output 500 frames in 39.67 seconds (12.60 fps)
The time variations could be related to the antivirus scanning my external USB disk in the background
Almost all group policies are just regedits with airs, and work the same on Home once set: This one is HKEY_CURRENT_USER\System\CurrentControlSet\Policies DWORD SeLockMemoryPrivilege value 0x1. Reboot after setting.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. ~ Ed Howdershelt
foxyshadis is offline   Reply With Quote
Old 17th April 2018, 16:35   #3052  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 421
I just tried the 4k code snipped by HolyWu + PS code above and it is SLOW:
Output 1000 frames in 172.86 seconds (5.78 fps) | 15% CPU load :-/

With "$env:VS_NO_LARGE_PAGES = 1"
Output 1000 frames in 24.06 seconds (41.55 fps) | 75% CPU load

p.s. I have 16gb of ram.
__________________
Search and denoise

Last edited by ChaosKing; 17th April 2018 at 18:31.
ChaosKing is online now   Reply With Quote
Old 17th April 2018, 22:33   #3053  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 421
Now with core.max_cache_size = 32768

Code:
Output 1000 frames in 16.40 seconds (60.96 fps)
$env:VS_NO_LARGE_PAGES = 1
Output 1000 frames in 16.24 seconds (61.59 fps)
From 40 to 60fps, nice speed up! CPU load was 100% now.
__________________
Search and denoise
ChaosKing is online now   Reply With Quote
Old 18th April 2018, 16:49   #3054  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 5,627
Quote:
I have 16gb of ram.
+
Quote:
with core.max_cache_size = 32768
this seems strange,...
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 18th April 2018, 22:06   #3055  |  Link
Stephen R. Savage
Registered User
 
Stephen R. Savage's Avatar
 
Join Date: Nov 2009
Posts: 337
Quote:
Originally Posted by Selur View Post
+

this seems strange,...
It is because the memory subsystem in VapourSynth needs tuning. When memory limit is 32 GB, only 3.2 GB can be allocated to the working set. The rest is reserved for filter caches.
Stephen R. Savage is offline   Reply With Quote
Old 19th April 2018, 17:24   #3056  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,402
I did some tests on a Blu-ray source with my usual script to denoise and downsize to 720p, 2000 frames from the middle:

v44-test1, cache 32768:
Large pages - 10,74 fps
No large pages - 11,06 fps

v43
cache 32768 - 11,02 fps
default cache - 10,02 fps

So no use for me, but the cache size needs to be increased - so thanks for that tip I have 16 GB of memory and an AMD Ryzen 1800X system.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 21st April 2018, 08:10   #3057  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 5,627
Still a bit confused about max_cache_size.
Quote:
max_cache_size

Set the upper framebuffer cache size after which memory is aggressively freed. The value is in megabytes.

set_max_cache_size(mb)

Deprecated, use max_cache_size instead.
so since max cache size is in MB and you have 16GB why set it to 32768 = 32GB, shouldn't your 16GB (- normal system memory usage) of RAM be the largest value one should set and not twice the available RAM or am I missing something?
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 25th April 2018, 16:34   #3058  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,380
Quote:
Originally Posted by Selur View Post
Still a bit confused about max_cache_size.

so since max cache size is in MB and you have 16GB why set it to 32768 = 32GB, shouldn't your 16GB (- normal system memory usage) of RAM be the largest value one should set and not twice the available RAM or am I missing something?
It's a workaround for large pages in the current version only. If you aren't interested in large pages right now, or you're worried about a filter sucking up all of the cache, don't touch it.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. ~ Ed Howdershelt
foxyshadis is offline   Reply With Quote
Old 25th April 2018, 16:52   #3059  |  Link
DJATOM
Registered User
 
Join Date: Sep 2010
Location: Ukraine, Bohuslav
Posts: 119
I found a bug. Well, not sure if that's really a bug, but might be unwanted behaviour of resizer.
Nowadays almost all anime blu-rays are upscales. I'd like to fight such upscales downscaling to original resolution using the mixed Descale and Spline36 clip, upscale back with eedi3 and merge it with source clip using lines mask.
Today my friend noticed slight shift in the source/AA comparisons. I tried to debug my script and found that resizer is just copying clip instead of fixing eedi3 shift. In that case 1920x1080 clip was downscaled to 960x540 and upscaled back (so we got 1920x1080 after eedi3 stuff).
As I understand, vapoursynth (zimg) resizer is just copying clip if input width and height matches with output. That might be unwanted if cropping options is set.
On my opinion resizer should check for crop options and do proper resizing in that case.
__________________
Me on GitHub | My Telegram
DJATOM is offline   Reply With Quote
Old 25th April 2018, 17:43   #3060  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 5,627
@foxyshadis: I get what max_cache_size is meant to do, what I don't get is why setting it to two times your RAM size is the right thing.
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Reply

Tags
speed, vaporware, vapoursynth

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 01:52.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.