Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 21st January 2019, 17:21   #21  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,636
Here's the one I used:
Code:
import vapoursynth as vs

core = vs.get_core()

orig = core.dgdecodenv.DGSource(r'O:\Testclips\test2.dgi', fulldepth=True)
orig = core.f3kdb.Deband(orig, preset="medium", output_depth=10)

clp = core.ffms2.Source(source=r'c:\x265\aq\aqmode3.hevc')
orig = core.resize.Bicubic(orig, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
clp = core.resize.Bicubic(clp, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)

result = core.vmaf.VMAF(orig, clp, model=1, log_path="c:\x265\aq\clp1.log", log_fmt=0, pool=1, ci=True)

result.set_output()
I also noticed that there's no logfile written if I run that one through "vspipe script.vpy ."
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 21st January 2019, 21:11   #22  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,636
The process also hangs without returning to the command prompt. The values are output but I need to use CTRL+C to get the prompt back.

C:\>vspipe c:\x265\aq\compare.vpy .
Start calculating VMAF score...
Exec FPS: 0.501642
VMAF score (harmonic_mean) = 98.925030

The Exec FPS must be a wrong value, the clips are 1385 frames long and it only took a couple of seconds to produce that output. If I ran with the -p option, I got those out of memory errors quite soon.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 21st January 2019, 21:34   #23  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,164
Quote:
Originally Posted by Boulder View Post

C:\>vspipe c:\x265\aq\compare.vpy .
Start calculating VMAF score...
Exec FPS: 0.501642
VMAF score (harmonic_mean) = 98.925030

The Exec FPS must be a wrong value, the clips are 1385 frames long and it only took a couple of seconds to produce that output.
Have you try running it with vs editor's benchmark? My 1000 frames 1080p anime took about a minute and 20 seconds, yours should be doubling or tripling my time comparing 4k clips.
lansing is offline   Reply With Quote
Old 21st January 2019, 21:40   #24  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,636
Just tried - it crashed by vanishing without a warning almost right after it started. I raised the cache size to 2048 MB and it hung at frame 30 which it probably what happens with vspipe as well.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 21st January 2019, 23:03   #25  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,164
Quote:
Originally Posted by Boulder View Post
Just tried - it crashed by vanishing without a warning almost right after it started. I raised the cache size to 2048 MB and it hung at frame 30 which it probably what happens with vspipe as well.
I tried your script, mine stop moving about frame 14 too. I try lowering the resize resolution to 3000x2000, it was able to run with RAM usage jumping between 3.8G to 4.8G. So it should be an insufficient memory problem?
lansing is offline   Reply With Quote
Old 22nd January 2019, 02:23   #26  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,042
No problems here. Max ram usage was 10gb.
Maybe it is also a path problem? I used log_path=r"D:\test\clp1.log"
The x265 folder exists on c? Bcs if not vapoursynth can't create one without admin rights.
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database || https://github.com/avisynth-repository
ChaosKing is offline   Reply With Quote
Old 22nd January 2019, 02:52   #27  |  Link
gonca
Registered User
 
Join Date: Jul 2012
Location: Scarborough, Canada
Posts: 947
Quote:
Originally Posted by Boulder View Post
Did I understand correctly that model 0 should be used if the tested material is 1080p or less? At first I thought that it meant the viewing device but after reading the FAQ, it looks like it's the content itself.
Could someone expand on this.
Is it the device or video?
Does the video have to be resized?
gonca is offline   Reply With Quote
Old 22nd January 2019, 03:09   #28  |  Link
HolyWu
Registered User
 
HolyWu's Avatar
 
Join Date: Aug 2006
Location: Taiwan
Posts: 662
I also can't reproduce the issues you two encountered with a 2000 frames 1080p upscaled to 4K, no matter using vspipe or vsedit. And not prefixing the string containing a single backslash with r is always a bad idea.
HolyWu is offline   Reply With Quote
Old 22nd January 2019, 03:44   #29  |  Link
HolyWu
Registered User
 
HolyWu's Avatar
 
Join Date: Aug 2006
Location: Taiwan
Posts: 662
Quote:
Originally Posted by gonca View Post
Could someone expand on this.
Is it the device or video?
Does the video have to be resized?
It's the viewing device. See https://github.com/Netflix/vmaf/blob...ible-artifacts.
HolyWu is offline   Reply With Quote
Old 22nd January 2019, 03:59   #30  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,164
I did another test, my PC has 16G RAM and I start the benchmark with all my apps closed dropping total ram usage to 3.5G. And now the benchmark ran without chocking with RAM usage for the process peaks at 6G rarely.

And then I did another benchmark with starting ram usage at 8.4G, this time the process stopped after the first few frames with its ram usage start from 200MB -> 3GB -> 500MB.

So this seems like a not-enough-memory issue, but with 16G, I should still have 1G excessive memory to run the second benchmark.
lansing is offline   Reply With Quote
Old 22nd January 2019, 04:50   #31  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,636
Quote:
Originally Posted by HolyWu View Post
And this seems to confirm that upscaling to that resolution is the way to go, although getting a VMAF score from an originally 1080p source on a 4K display is not optimal.

Code:
If, say, for a distorted video of 480 resolution, we still want to predict its quality viewing from 3 times the height (not 6.75), how can this be achieved?

    If the 480 distorted video comes with a source (reference) video of 1080 resolution, then the right way to do it is to upsample the 480 video to 1080, and calculate the VMAF at 1080, together with its 1080 source.

    If the 480 distorted video has only a 480 reference, then you can still upsample both distorted/reference to 1080, and calculate VMAF. A caveat is, since the VMAF model was not trained with upsampled references,
the prediction would not be as accurate as 1).
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 22nd January 2019, 04:56   #32  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,636
Quote:
Originally Posted by lansing View Post
I did another test, my PC has 16G RAM and I start the benchmark with all my apps closed dropping total ram usage to 3.5G. And now the benchmark ran without chocking with RAM usage for the process peaks at 6G rarely.

And then I did another benchmark with starting ram usage at 8.4G, this time the process stopped after the first few frames with its ram usage start from 200MB -> 3GB -> 500MB.

So this seems like a not-enough-memory issue, but with 16G, I should still have 1G excessive memory to run the second benchmark.
When I first tested things, I didn't have any other big processes running and I had no cache size restrictions in the script. The memory usage jumped to around 8GB and still got the out of memory error. Task Manager showed that the total memory usage was around 65-70% when it happened, so there was still more memory available but maybe Vapoursynth restricted it.

It looks like I cannot run an encode simultaneously because it seems end abruptly quite often when the other VS process grabs hold of most of the memory on the computer. I'm getting 8GB more this week which is good for testing this issue.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 22nd January 2019, 23:38   #33  |  Link
gonca
Registered User
 
Join Date: Jul 2012
Location: Scarborough, Canada
Posts: 947
Doing a test run
4K display, 4K sample
Using VDub2 64 bit for playback of script
So far it is slow and CPU is at near 100%
RAM for VDub2 is +/- 12 GB, but overall I see spikes to near 16GB overall, VDub2 has spiked to about 14GB on some occasions.
When this run is over I will try VSEditor
gonca is offline   Reply With Quote
Old 23rd January 2019, 00:17   #34  |  Link
gonca
Registered User
 
Join Date: Jul 2012
Location: Scarborough, Canada
Posts: 947
VDub2 seems to run fine
Tried VSEditor and got the out of memory error early
Added one line and it seems to go through fine
Numbers (RAM) are about the same as VDub2

Original script

Code:
import vapoursynth as vs
core = vs.get_core()
core.std.LoadPlugin("C:/Program Files (Portable)/dgdecnv/x64 Binaries/DGDecodeNV.dll")
core.std.LoadPlugin(r'C:\Users\LUIS\Desktop\VMAF-r2\plugins64\VMAF.dll')
orig = core.dgdecodenv.DGSource(r'I:\original.dgi', fieldop=0, fulldepth=True)
orig = core.resize.Point(orig, format=vs.YUV420P10)
dist = core.dgdecodenv.DGSource(r'I:\distorted.dgi', fieldop=0, fulldepth=True)
dist = core.resize.Point(dist, format=vs.YUV420P10)
result = core.vmaf.VMAF(orig, dist, model=1, log_path=r'I:\log.log', log_fmt=1, pool=1, ci=True)
result.set_output ()
Modified script

Code:
import vapoursynth as vs
core = vs.get_core()
core.max_cache_size = 16384
core.std.LoadPlugin("C:/Program Files (Portable)/dgdecnv/x64 Binaries/DGDecodeNV.dll")
core.std.LoadPlugin(r'C:\Users\LUIS\Desktop\VMAF-r2\plugins64\VMAF.dll')
orig = core.dgdecodenv.DGSource(r'I:\original.dgi', fieldop=0, fulldepth=True)
orig = core.resize.Point(orig, format=vs.YUV420P10)
dist = core.dgdecodenv.DGSource(r'I:\distorted.dgi', fieldop=0, fulldepth=True)
dist = core.resize.Point(dist, format=vs.YUV420P10)
result = core.vmaf.VMAF(orig, dist, model=1, log_path=r'I:\log.log', log_fmt=1, pool=1, ci=True)
result.set_output ()
Seems minimum amount of RAM for 4K is 16GB + all background processes

Last edited by gonca; 23rd January 2019 at 00:39. Reason: typo
gonca is offline   Reply With Quote
Old 25th January 2019, 04:53   #35  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,164
I'm wondering, right now with my machine I'm doing 12 fps on 1440x1080 clips with 3G ram usage. If I'm to upgrade to a faster machine that do 3 times this speed, will my ram usage also go up by 3 times?
lansing is offline   Reply With Quote
Old 25th January 2019, 06:47   #36  |  Link
HolyWu
Registered User
 
HolyWu's Avatar
 
Join Date: Aug 2006
Location: Taiwan
Posts: 662
Quote:
Originally Posted by lansing View Post
I'm wondering, right now with my machine I'm doing 12 fps on 1440x1080 clips with 3G ram usage. If I'm to upgrade to a faster machine that do 3 times this speed, will my ram usage also go up by 3 times?
I guess the ram usage should only be relevant to the video resolution, bit depth, and number of threads.
HolyWu is offline   Reply With Quote
Old 25th January 2019, 21:37   #37  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,164
I did another test on this, I disabled the core in task manager and ran benchmark on 1 to 8 threads, speed rises as more threads were used, but the ram usage are the same on the all of them.

A side note, setting the thread number in the script doesn't work
Code:
core = vs.get_core(threads=1)
lansing is offline   Reply With Quote
Old 26th January 2019, 03:08   #38  |  Link
HolyWu
Registered User
 
HolyWu's Avatar
 
Join Date: Aug 2006
Location: Taiwan
Posts: 662
Quote:
Originally Posted by lansing View Post
A side note, setting the thread number in the script doesn't work
Code:
core = vs.get_core(threads=1)
Ah, libvmaf uses pthreads for its internal multithreading, and I didn't set it according to core's number of threads but leave it at its default which uses all threads available. I'll fix it at the next release.
HolyWu is offline   Reply With Quote
Old 1st February 2019, 06:01   #39  |  Link
HolyWu
Registered User
 
HolyWu's Avatar
 
Join Date: Aug 2006
Location: Taiwan
Posts: 662
Update r3.
  • Update libvmaf to v1.3.13, which includes performance improvement.
  • Change default pool to 1.
  • Set libvmaf's threads according to core's number of worker threads.
HolyWu is offline   Reply With Quote
Old 1st February 2019, 19:28   #40  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,164
I couldn't update it through vsrepo?
lansing is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:58.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.