Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 21st March 2019, 04:05   #221  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by Iron_Mike View Post
good news is that seekmode=0 is rock solid on my end (and not slower), and Lsmash as well, so it's not a show stopper...
If you ignore the long decimal place variation issue , that you reported here with seekmode=0 . That's not a deal breaker, I guess you could "round" it or truncate it . So still some weirdness going on there than I cannot replicate

https://forum.doom9.org/showthread.p...88#post1869188



And also , remuxing to MKV "fixed" it for me and CK with default seekmode, but not you , is that correct ? So some weirdness and inconsistencies between setups too


I always have like 15 different ffms2 versions handy for AVS and VPY. The way I organize it is keep each one in a separate labelled subfolder. Because each build has different kinds of quirks and bugs . Some with different formats, e.g some drop certain types of MOV support, others will get the fps slightly off with buggy timecodes. Others will double TS frames (reading field rate as frame rate) . Some will have VP9 issues. Some AV1 issues. Some will have seek issues with certain formats. LSmash versions can share these little bugs too , but they are not as frequently released

I would love a "super duper" ffms2 and lsmash version
poisondeathray is offline   Reply With Quote
Old 21st March 2019, 04:15   #222  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
Quote:
Originally Posted by Iron_Mike View Post
well... that took a while.
You just don't stop do you.

Quote:
Originally Posted by WorBry View Post
Quote:
Originally Posted by Iron_Mike View Post
So there is an issue, and clearly this is of interest for others, who may run into the same problems...
Not denying there is an issue, but I'm not seeing it.
__________________
Nostalgia's not what it used to be
WorBry is offline   Reply With Quote
Old 21st March 2019, 04:23   #223  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
Quote:
Originally Posted by poisondeathray View Post
If you ignore the long decimal place variation issue , that you reported here with seekmode=0 . That's not a deal breaker, I guess you could "round" it or truncate it . So still some weirdness going on there than I cannot replicate

https://forum.doom9.org/showthread.p...88#post1869188
Nor I.

Quote:
Originally Posted by poisondeathray View Post
LSmash versions can share these little bugs too...
Quote:
Originally Posted by WorBry View Post
What about LWLibavSource ?
Did you test LWLibavSource also ?
__________________
Nostalgia's not what it used to be

Last edited by WorBry; 21st March 2019 at 04:29.
WorBry is offline   Reply With Quote
Old 21st March 2019, 04:35   #224  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by WorBry View Post

Did you test LWLibavSource also ?
Not on this series


And for MP4/MOV container , you can use LibavSMASHSource without indexing . But that might affect the seeking here , might be worth while to test
poisondeathray is offline   Reply With Quote
Old 21st March 2019, 04:36   #225  |  Link
Iron_Mike
Registered User
 
Join Date: Jul 2010
Posts: 132
Quote:
Originally Posted by poisondeathray View Post
If you ignore the long decimal place variation issue , that you reported here with seekmode=0 . That's not a deal breaker, I guess you could "round" it or truncate it . So still some weirdness going on there than I cannot replicate
yeah, in some cases it provided more precision, in a perfectly stable, repeatable system that should not happen

Quote:
Originally Posted by poisondeathray View Post
And also , remuxing to MKV "fixed" it for me and CK with default seekmode, but not you , is that correct ?
I've not done that yet, if you tell me exactly which tools involved I can test it... or can I just copy the streams over into an mkv container via ffmpeg ?
Iron_Mike is offline   Reply With Quote
Old 21st March 2019, 04:40   #226  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by Iron_Mike View Post
I've not done that yet, if you tell me exactly which tools involved I can test it... or can I just copy the streams over into an mkv container via ffmpeg ?

I used mkvmerge (mkvtoolnix is the GUI)

CK reported this finding here
https://forum.doom9.org/showthread.p...33#post1869333

Sorry I got mixed up; it was WorBry that still got seek errors after remuxing to MKV with several methods
poisondeathray is offline   Reply With Quote
Old 21st March 2019, 04:51   #227  |  Link
Iron_Mike
Registered User
 
Join Date: Jul 2010
Posts: 132
@ChaosKing

could you assist in putting together a stripped down, minimal venv for a dedicated ffms2 test ?

In the vein of your Fatpack just extremely slimmed down - only to read vid files via ffms2... so:

> Py 3.7
> VS
> ffms2 Wolfberry
> all ffms2 dependencies

this should not be much more than 120MB... we could zip it and distrbiute it for testing...

read in your thread that you simply "threw everything in one folder", so if u lmk any caveats or your problems I can do it or maybe you can do it...

or can we just remove all plugs from your latest Fatpack and use that as a base... ?
Iron_Mike is offline   Reply With Quote
Old 21st March 2019, 04:54   #228  |  Link
Wolfberry
Helenium(Easter)
 
Wolfberry's Avatar
 
Join Date: Aug 2017
Location: Hsinchu, Taiwan
Posts: 99
Quote:
Originally Posted by poisondeathray View Post
I always have like 15 different ffms2 versions handy for AVS and VPY. The way I organize it is keep each one in a separate labelled subfolder. Because each build has different kinds of quirks and bugs . Some with different formats, e.g some drop certain types of MOV support, others will get the fps slightly off with buggy timecodes. Others will double TS frames (reading field rate as frame rate) . Some will have VP9 issues. Some AV1 issues. Some will have seek issues with certain formats. LSmash versions can share these little bugs too , but they are not as frequently released

I would love a "super duper" ffms2 and lsmash version
Quote:
Originally Posted by qyot27 View Post
Read: it may have nothing to do with the encoder or the decoder, but about the ability of FFMS2 to use the FFmpeg API to parse the bitstream in the correct way, and certain encoder/decoder combinations expose errant behavior in the parser. This would explain why ffplay (or mpv) works fine on the same sample and the same build of libdav1d that FFMS2 uses, but FFMS2 exhibits problems.
I think that the reason behind the different kinds of quirks and bugs is the different FFmpeg version that FFMS2 / L-SMASH is built with (like 3.x or 4.x)

A newer FFmpeg version can fix/add support for some formats but can also introduce new bugs in some other formats in FFMS2 / L-SMASH due to their ability to use the FFmpeg API to parse the bitstream in the correct way.
__________________
Monochrome Anomaly
Wolfberry is offline   Reply With Quote
Old 21st March 2019, 04:55   #229  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
Quote:
Originally Posted by poisondeathray View Post
And for MP4/MOV container , you can use LibavSMASHSource without indexing . But that might affect the seeking here , might be worth while to test
https://forum.doom9.org/showthread.p...32#post1869332
__________________
Nostalgia's not what it used to be
WorBry is offline   Reply With Quote
Old 21st March 2019, 04:57   #230  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
Quote:
Originally Posted by poisondeathray View Post
it was WorBry that still got seek errors after remuxing to MKV with several methods
With the seek-test script yes, but not in the metric tests.
__________________
Nostalgia's not what it used to be
WorBry is offline   Reply With Quote
Old 21st March 2019, 05:21   #231  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by WorBry View Post
With the seek-test script yes, but not in the metric tests.
But for Iron_Mike, didn't he say the seek-test correlated with the other metric tests? The gist I got was that it was predictive for him - when one one failed, the other failed ?

Basically you guys were getting something different from those results compared to each other . And I'm getting a 3rd different

Seek test fails both MP4 and MKV with default seekmode, but metrics inconsistent with MP4, but consistent with MKV



Another question is why does this vpy script request frames out of order, instead of linearly? You can use the same decoder, the same ffms2 source filter, and use -f vapoursynth in Wolfberry's ffmpeg and the ssim / psnr there and it's consistent 10/10 times because frames are requested linearly in order . I realize from WorBry's earlier discussion the ssim calcs are slightly different; but ideally this should be a linear operation.

In terms of difficulty the synthetic seek test is like a torture, worse case sceneario .
poisondeathray is offline   Reply With Quote
Old 21st March 2019, 06:38   #232  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
Quote:
Originally Posted by poisondeathray View Post
Quote:
Originally Posted by WorBry View Post
With the seek-test script yes, but not in the metric tests.
But for Iron_Mike, didn't he say the seek-test correlated with the other metric tests? The gist I got was that it was predictive for him - when one one failed, the other failed ?

Basically you guys were getting something different from those results compared to each other .
Right, and this is why I looked at the alignment of the reference and test file output frames when applying random frame searches vs playback from frame #0 in VSEditor:

https://forum.doom9.org/showthread.p...23#post1869323

Quote:
Originally Posted by WorBry View Post
So it is only random seeking with ffms2 (default, seekmode=1), which is what the seek-test script does, that gives these frame offsets, not sequential output from frame #0, as is the case when the metric test scripts are run with Benchmark to generate the score log files.

This perfectly explains why the seek-test script gave errors with the encoded x265 file and ffms2 mode #1 (seekmode=1), yet the results of the SSIM:GMSD metric tests with ffms2 (default, seekmode=1) were in perfect agreement (aggregate and per frame scores) with those produced with LWLibavSource - in my tests at least.
Quote:
Originally Posted by poisondeathray View Post
And I'm getting a 3rd different

Seek test fails both MP4 and MKV with default seekmode, but metrics inconsistent with MP4, but consistent with MKV
The difference between me and thee being that I also get consistent metrics with mp4 and ffms2 in default seekmode=1
__________________
Nostalgia's not what it used to be
WorBry is offline   Reply With Quote
Old 21st March 2019, 06:42   #233  |  Link
Iron_Mike
Registered User
 
Join Date: Jul 2010
Posts: 132
Quote:
Originally Posted by poisondeathray View Post
But for Iron_Mike, didn't he say the seek-test correlated with the other metric tests? The gist I got was that it was predictive for him - when one one failed, the other failed ?
correct, the seek script predicted things correctly for me.


alright, copied video stream of the mp4 encode w/ custom keyframes and open-gop into mkv container via ffmpeg

Code:
<ffmpeg> -i encode_with_keyfranmes_open_gop.mp4 -c copy encode_with_keyfranmes_open_gop.mkv
(lmk if that is not the correct way to create the mkv file)


> 1st metrics test run: MDSI - mkv - seekmode=1

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 202.9107845033357604980217559

> 2nd metrics test run: MDSI - mkv - seekmode=1 (same script)

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986

> 3rd metrics test run: MDSI - mkv - seekmode=1 (same script, just more reps)

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986

> 4th metrics test run: MDSI - mkv - seekmode=1 (same script, just more reps)

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 194.8927784337109077039684736
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986

> 5th metrics test run: GMSD/SSIM - mkv - seekmode=1

stop 105.8009960111949401551090232 321.3940602997202934110188006
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 114.6468917102554510301093416 275.2515876977237659073427521
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524


> seek-test.py from CK, test #1 on mkv: seek errors, 3 frames off



> the corresponding .mp4 file (encode w/ custom keyframes and open-gop) - seekmode=1

> MDSI:

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986


> GMSD/SSIM:

stop 123.9634227532677362149104996 227.1636231975790896908673488
stop 122.1886438717397119491092642 235.7664674057195217937721310
stop 123.9120154727625848778416187 228.0649240752797068665369550
stop 123.6926367223807698425552817 228.4161432110821759733365609
stop 123.7779981257600182276146230 228.0649240752797068665369550



--> so why is the MDSI metric much more stable (although not perfect) w/ seekmode=1 ?
Iron_Mike is offline   Reply With Quote
Old 21st March 2019, 09:37   #234  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
Quote:
As I posted before, this old ffms2 version crashes for me w/ your Fatpack... I simply copied the x64 plug into the Fatpack's VS plugins folder... anything else I need to do ?
Nope, all you need is to replace the "old" ffms2.dll. You probably missing a vs runtime.
You could try downloading the 64 bit version of these files: https://www.dll-files.com/msvcp_win.dll.html, https://www.dll-files.com/msvcrt.dll.html and see if it helps. Copy the runtime dlls also inside the plugins folder where ffms2 is located.
And maybe all msvcp*.dlls from VapourSynth64 folder just in case But I think msvcrt.dll should fix it.

Quote:
Originally Posted by poisondeathray View Post

I always have like 15 different ffms2 versions handy for AVS and VPY. The way I organize it is keep each one in a separate labelled subfolder. Because each build has different kinds of quirks and bugs . Some with different formats, e.g some drop certain types of MOV support, others will get the fps slightly off with buggy timecodes. Others will double TS frames (reading field rate as frame rate) . Some will have VP9 issues. Some AV1 issues. Some will have seek issues with certain formats. LSmash versions can share these little bugs too , but they are not as frequently released

I would love a "super duper" ffms2 and lsmash version
How about a test folder with some video files and a folder with different ffms2/lsmash dlls + a test script wich just tests all files with all dlls via seek-test? Like an automated mass seek-test if you will. Easy to make... the results could be written into a txt or csv file.

Quote:
Originally Posted by Iron_Mike View Post
@ChaosKing

could you assist in putting together a stripped down, minimal venv for a dedicated ffms2 test ?

In the vein of your Fatpack just extremely slimmed down - only to read vid files via ffms2... so:

> Py 3.7
> VS
> ffms2 Wolfberry
> all ffms2 dependencies

this should not be much more than 120MB... we could zip it and distrbiute it for testing...

read in your thread that you simply "threw everything in one folder", so if u lmk any caveats or your problems I can do it or maybe you can do it...

or can we just remove all plugs from your latest Fatpack and use that as a base... ?
Sure why not. Will upload it a bit later.


Quote:
Another question is why does this vpy script request frames out of order, instead of linearly?
Because the requests are made in parallel wich makes your script run faster.
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database
ChaosKing is offline   Reply With Quote
Old 21st March 2019, 10:57   #235  |  Link
Wolfberry
Helenium(Easter)
 
Wolfberry's Avatar
 
Join Date: Aug 2017
Location: Hsinchu, Taiwan
Posts: 99
Quote:
Originally Posted by ChaosKing View Post
You could try downloading the 64 bit version of these files: https://www.dll-files.com/msvcp_win.dll.html, https://www.dll-files.com/msvcrt.dll.html and see if it helps. Copy the runtime dlls also inside the plugins folder where ffms2 is located.
And maybe all msvcp*.dlls from VapourSynth64 folder just in case But I think msvcrt.dll should fix it.
Well, I don't think downloading arbitrary runtime dlls is a good way to solve runtime dependency problems.

If you are lazy you can just install an AIO pack as mentioned in this thread.
__________________
Monochrome Anomaly

Last edited by Wolfberry; 21st March 2019 at 15:20.
Wolfberry is offline   Reply With Quote
Old 21st March 2019, 11:55   #236  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
Correct. It's just to quickly confirm that this dll is the culprit. I checked with ldd and dependency walker and msvcrt stood out. Or maybe it's ADVAPI32.dll but I think its a system dll.

EDIT
ok lol, the "old" ffms2 crashes with the y4m file!!! I only tested it via seek-test and the mp4 file in vsedit.

Code:
ldd ffms2.dll <--- old ffms2.dll
        ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffd32440000)
        KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x7ffd2f980000)
        KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x7ffd2e5c0000)
        apphelp.dll => /cygdrive/c/WINDOWS/SYSTEM32/apphelp.dll (0x7ffd2c780000)
        ADVAPI32.dll => /cygdrive/c/WINDOWS/System32/ADVAPI32.dll (0x7ffd2fa60000)
        msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x7ffd2fe20000)
        sechost.dll => /cygdrive/c/WINDOWS/System32/sechost.dll (0x7ffd2f8e0000)
        RPCRT4.dll => /cygdrive/c/WINDOWS/System32/RPCRT4.dll (0x7ffd30380000)
        WS2_32.dll => /cygdrive/c/WINDOWS/System32/WS2_32.dll (0x7ffd323a0000)
        USER32.dll => /cygdrive/c/WINDOWS/System32/USER32.dll (0x7ffd2fb20000)
        Secur32.dll => /cygdrive/c/WINDOWS/SYSTEM32/Secur32.dll (0x7ffd1c190000)
        win32u.dll => /cygdrive/c/WINDOWS/System32/win32u.dll (0x7ffd2f410000)
        GDI32.dll => /cygdrive/c/WINDOWS/System32/GDI32.dll (0x7ffd32370000)
        gdi32full.dll => /cygdrive/c/WINDOWS/System32/gdi32full.dll (0x7ffd2e9d0000)
        msvcp_win.dll => /cygdrive/c/WINDOWS/System32/msvcp_win.dll (0x7ffd2eb70000)
        ucrtbase.dll => /cygdrive/c/WINDOWS/System32/ucrtbase.dll (0x7ffd2e8b0000)
        SSPICLI.DLL => /cygdrive/c/WINDOWS/SYSTEM32/SSPICLI.DLL (0x7ffd2e340000)
        IMM32.DLL => /cygdrive/c/WINDOWS/System32/IMM32.DLL (0x7ffd30510000)

Code:
$ ldd ffms2_.dll <-- (dll by Wolfberry)
        ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffd32440000)
        KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x7ffd2f980000)
        KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x7ffd2e5c0000)
        bcrypt.dll => /cygdrive/c/WINDOWS/System32/bcrypt.dll (0x7ffd2f3e0000)
        msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x7ffd2fe20000)
        USER32.dll => /cygdrive/c/WINDOWS/System32/USER32.dll (0x7ffd2fb20000)
        Secur32.dll => /cygdrive/c/WINDOWS/SYSTEM32/Secur32.dll (0x7ffd1c190000)
        win32u.dll => /cygdrive/c/WINDOWS/System32/win32u.dll (0x7ffd2f410000)
        GDI32.dll => /cygdrive/c/WINDOWS/System32/GDI32.dll (0x7ffd32370000)
        gdi32full.dll => /cygdrive/c/WINDOWS/System32/gdi32full.dll (0x7ffd2e9d0000)
        msvcp_win.dll => /cygdrive/c/WINDOWS/System32/msvcp_win.dll (0x7ffd2eb70000)
        ucrtbase.dll => /cygdrive/c/WINDOWS/System32/ucrtbase.dll (0x7ffd2e8b0000)
        WS2_32.dll => /cygdrive/c/WINDOWS/System32/WS2_32.dll (0x7ffd323a0000)
        RPCRT4.dll => /cygdrive/c/WINDOWS/System32/RPCRT4.dll (0x7ffd30380000)
        SSPICLI.DLL => /cygdrive/c/WINDOWS/SYSTEM32/SSPICLI.DLL (0x7ffd2e340000)
        sechost.dll => /cygdrive/c/WINDOWS/System32/sechost.dll (0x7ffd2f8e0000)
        IMM32.DLL => /cygdrive/c/WINDOWS/System32/IMM32.DLL (0x7ffd30510000)
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database

Last edited by ChaosKing; 21st March 2019 at 12:05.
ChaosKing is offline   Reply With Quote
Old 21st March 2019, 19:20   #237  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
Quote:
Originally Posted by WorBry View Post
With the seek-test script yes, but not in the metric tests.
Just to mention that the seek-test script (Wolfberry ffms2 seekmode=1) reports no issues with x264 mp4/mkv

Code:
ffmpeg -i  {Path}:/crowd_run_1080p50.y4m -vcodec libx264 -preset slow -crf 28 -pix_fmt yuv420p -r 50/1 -x264-params colorprim=bt709:transfer=bt709:colormatrix=bt709 {Path}:/crowd_run_1080p50_y4m_x264_CRF28.mp4
The rav1e.mkv encodes of Crowd Run that ChaosKing uploaded earlier...

https://forum.doom9.org/showthread.p...94#post1866894

....do give errors (1 frame off) with the seek-test (ffms2 seekmode=1) but the SSIM:GMSD results are consistent on consecutive runs and the log files give the frames in correct order.
__________________
Nostalgia's not what it used to be

Last edited by WorBry; 22nd March 2019 at 15:20.
WorBry is offline   Reply With Quote
Old 22nd March 2019, 07:40   #238  |  Link
Iron_Mike
Registered User
 
Join Date: Jul 2010
Posts: 132
Quote:
Originally Posted by Iron_Mike View Post
ran a couple of tests to see how preset behave for x265...

here I tested medium vs. fast... same range as WorBry CRF 0-30, used the same graph layout

more stats - x265 - preset: medium vs. fast


Butteraugli / MDSI per CRF



VMAF / SSIM / GMSD per bitrate (Mbps)



Butteraugli / MDSI per bitrate (Mbps)


Last edited by Iron_Mike; 24th March 2019 at 10:19.
Iron_Mike is offline   Reply With Quote
Old 22nd March 2019, 19:38   #239  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
Incidentally, in your tests, what results do you get for the different metrics when you perform a 'self test' on your x265 encodes i.e. use the same encode as the reference and test clip as a positive control ?

VMAF is an exception because it gives a component 'motion2' metric score of 0 to the first frame which can skew the per-frame and aggregate VMAF score.

But for the SSIM, GMSD, MDSI and Butteraugli tests, the 'self' scores should be absolute i.e. 1 for SSIM and 0 for the other three. If not, something is amiss.

I ran self tests in all of the studies I carried out earlier, as well as duplicate tests on representative encodes (typically at the highest and lowest quality settings), and encode duplicates, (except for the rav1e series, which I didn't encode) to check for reproducibility before proceeding to test the entire series. And found no inconsistencies.
__________________
Nostalgia's not what it used to be

Last edited by WorBry; 22nd March 2019 at 20:20.
WorBry is offline   Reply With Quote
Old 22nd March 2019, 22:32   #240  |  Link
Iron_Mike
Registered User
 
Join Date: Jul 2010
Posts: 132
Quote:
Originally Posted by WorBry View Post
Incidentally, in your tests, what results do you get for the different metrics when you perform a 'self test' on your x265 encodes i.e. use the same encode as the reference and test clip as a positive control ?
But for the SSIM, GMSD, MDSI and Butteraugli tests, the 'self' scores should be absolute i.e. 1 for SSIM and 0 for the other three. If not, something is amiss.
so run a control test on a x265 encode that has custom keyframes w/ open-gop ? seekmode=0 or seekmode=1 ? or what are we looking for here ?


Quote:
Originally Posted by WorBry View Post
VMAF is an exception because it gives a component 'motion2' metric score of 0 to the first frame which can skew the per-frame and aggregate VMAF score.
the "first frame" thing is only specific to the crowdrun file, we discussed this a while ago. When I ran extensive VMAF control tests on other files than crowdrun, the score was never 99.999 and it wasn't just the first frame score that threw it off...

as you know, NF does state that control test score of 98.x are to be expected...
Iron_Mike is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:04.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.