GMSD and SSIM Quality Metrics

[WorBry]

Originally Posted by ChaosKing

But have you also used the fmpeg Zeranoe 20190312-d227ed5 build?

It's the other way round. He used Zeranoe 20190312-d227ed5 to replicate my test procedure:

https://forum.doom9.org/showthread.p...94#post1869094

Originally Posted by Iron_Mike

As a test to match WorBrys results, I made two encodes - in both cases I used the same source (crowdrun 1080p50), the same ffmpeg Zeranoe build, and the same ffmpeg settings that WorBry used.

I get consistent results on serial testing. He get's these dodgy inconsistent results. How so ?

[WorBry]

Originally Posted by Iron_Mike

...it is the x265 KEYFRAMES parameter that creates an encode that will throw off seekmode...

Originally Posted by WorBry

...Nor have I run any tests with x265 encoded with custom key frame and rc-lookahead settings, because I have no interest in doing so

Well I have now the run tests with x265 CRF28 encoded with custom key frame interval, exactly as per your script:

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

And these are the results I get on serial testing with SSIM:GMSD (Downsample=False):

LWLibavSource:

stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 

ffms2- Wolfberry (default, i.e. seekmode=1)

stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786

No inconsistencies at all. And the same results regardless of whether the ffindex and lwi files from the first test in each series were retained or a fresh index file was generated for each test.

Compare that to your results.

Originally Posted by Iron_Mike

(3) encode w/ keyint, min-keyint but w/o rc-lookahead:

--> wanted to see if it is specifically the rc-lookahead setting that causes the issues w/ seekmode=1

ffmpeg command

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

Lsmash:
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165

FFMS2 (Wolfberry) - seekmode=0:
stop 83.19534706841786 435.99382191599165 
stop 83.19534706841786 435.99382191599165 
stop 83.19534706841786 435.99382191599165 
stop 83.19534706841786 435.99382191599165
stop 83.19534706841787 435.99382191599165

FFMS2 (Wolfberry) - seekmode=1 (default):
stop 108.17858520287257 307.76386120454794 
stop 124.03602892099701 228.48869961962274 
stop 83.19534706841786 435.99382191599165
stop 122.13314288222833 236.93921207380862 
stop 123.97733586255691 229.28604315863697

Also, as before, I extracted the log file data into LibreOffice Calc to run difference checks on the per-frame scores and I could find zero differences at all between the serial tests and between the LWLibavSource and ffms2 test series. And the frames were listed in correct sequence (0-499) in the log files.

Go figure.

[poisondeathray]

What hardware were you guys running on ?

It might partially have to do with threads and cores ; The more threads, the more requests, the higher chance of frame mismatches and seek errors if you're not using a robust seek method or indexing like dgsource or seekmode=0, threads=1

CK's seektest is like wild random seeks to simulate very bad case; but usually "normal" encoding scenarios , "normal scripts", even "normal" metric testing occurs in linear order .


But small inconsistencies, like 7th decimal place etc... suggest something else wrong. Unless you have duplicate frames in the test sequence (and the mild variation is just lossy encoding differences of duplicate frames), there is something else going on. I would start to look at other things, hardware issues, run memory diagnostics

[WorBry]

Originally Posted by poisondeathray

What hardware were you guys running on ?

Actually, I've been running these tests on an older PC (6-core AMD FX-6300 Vishera 3500 Mhz, NVidia GeForce GT730, Win10 x64) that I keep for grunt encoding/processing and linux stuff (dual boot). A bit sluggish with 4K , but it gets there. For the metric tests I've let it run on 6 threads.

Originally Posted by poisondeathray

But small inconsistencies, like 7th decimal place etc... suggest something else wrong. Unless you have duplicate frames in the test sequence (and the mild variation is just lossy encoding differences of duplicate frames), there is something else going on. I would start to look at other things, hardware issues, run memory diagnostics

There are no duplicate frames in the 10 sec Crowd Run sequence.

[poisondeathray]

Note the files you guys encoded are going to be slightly different too. Even with the same commandline . Because of encoding threads. Unless you guys have the same core and thread count on your hardware, or you explictly set threads

[ChaosKing]

I added Decimal() for testing purposes https://github.com/theChaosCoder/zoptilib
The metric values are now much longer, sum should always be the same between runs. Please test.

[Iron_Mike]

Originally Posted by WorBry

Yes, I read your post in great detail. And did you read mine ? I only referred to your first set of data with the x265 CRF26 created with the command line I gave you. "Simply use the provided ffmpeg commands in my post to re-create the problem" indeed - you've got a nerve. Yes I did, and I cannot replicate your inconsistent results - I get the exact same scores every time and a marginally higher SSIM score to you.

well, if you read the post then you may wanna allude to the point that the keyframes encode setting does not trip up seekmode on your system, b/c clearly as I outlined, the keyframes setting is the only differentiating factor in the encodes I made.

if you followed exactly the commands I posted and used the exact same sw packages I posted and linked to, then it may be a system difference, as poisondeathray pointed to. I asked you pages ago, what OS you're using...

But, point remains:
I can consistently repeat the problem w/ seekmode=1 (and the keyframes encode).
CK, who repeated the procedure from my post, ended up experiencing the exact same issues on his system.

So there is an issue, and clearly this is of interest for others, who may run into the same problems...

btw, this is not a ffmpeg specific version problem, I tested w/ three different ffmpeg versions..

[Iron_Mike]

Originally Posted by WorBry

Well I have now the run tests with x265 CRF28 encoded with custom key frame interval, exactly as per your script

did you use the download of crowd_run_1080p50.y4m or did you use your own encode from the 2160p version as ref file ?

[Iron_Mike]

Originally Posted by WorBry

Do you get the same dodgy results retaining the ffindex and lwi index files from the first test in a series vs creating a fresh index file for each ?

tried both ways, did not make a difference w/ seekmode=1 on the keyframes encoded file...

[WorBry]

Originally Posted by Iron_Mike

if you followed exactly the commands I posted and used the exact same sw packages I posted and linked ..

Good grief. You mean the sw packages I pointed you to in the first place.

Originally Posted by Iron_Mike

I asked you pages ago, what OS you're using...

Originally Posted by WorBry

Actually, I've been running these tests on an older PC (6-core AMD FX-6300 Vishera 3500 Mhz, NVidia GeForce GT730, Win10 x64) that I keep for grunt encoding/processing and linux stuff (dual boot).

And what OS and hardware are you running the tests on ?

Originally Posted by Iron_Mike

So there is an issue, and clearly this is of interest for others, who may run into the same problems...

Not denying there is an issue, but I'm not seeing it.

[WorBry]

Originally Posted by Iron_Mike

did you use the download of crowd_run_1080p50.y4m or did you use your own encode from the 2160p version as ref file ?

The crowd_run_1080p50.y4m file of course

[zorr]

Mike, can you take a look at what the log files look like when you get the inconsistent results (like below)?

FFMS2 (Wolfberry) - seekmode=1 (default):
stop 123.93944837285996 227.9372376995322
stop 123.74699115224307 228.86426005799092
stop 82.89543157743563 436.641839735243 
stop 121.66415998563879 239.07092253508384
stop 122.1422683865401 236.40679167570886

The error there is huge and it might be revealing to see what the scores are for the individual frames. Another option would be to save the resulting video with parameter showstats = True.

[Iron_Mike]

Originally Posted by WorBry

Not denying there is an issue, but I'm not seeing it.

another pointer is that, on my end, when running CK's test script option #1 and option #5 showed both success if the file was NOT encoded w/ keyframes setting... if it was encoded w/ keyframes, then option #1 (seekmode=0) showed seek errors, always 3 frames off...

meaning: seek errors on my system are repeatable (if triggered), and are both confirmed by CK's test script and the inconsistent GMSD/SSIM results.

you posted that you also had seek errors w/ option #1, yet when specifically setting seekmode=1 when running the GMSD/SSIM tests you get no errors...

[Iron_Mike]

Originally Posted by zorr

Mike, can you take a look at what the log files look like when you get the inconsistent results (like below)?

The error there is huge and it might be revealing to see what the scores are for the individual frames. Another option would be to save the resulting video with parameter showstats = True.

does that require the latest Zoptilib ?

where do I pass in the showstats parameter ? when I create an instance of the class, e.g.

zopti = Zopti(log_fp, metrics=metrics_list, showstats=True)

is that correct ?

[ChaosKing]

@Iron_Mike, yes it's correct.

Originally Posted by zorr

The error there is huge and it might be revealing to see what the scores are for the individual frames. Another option would be to save the resulting video with parameter showstats = True.

showstats doesn't affect the score since the text overlay is done after the ssim gsm filters. Plus I just tested it again, just to be sure

alt_clip = muv.SSIM(alt_clip, clip, **filter_args).text.FrameProps()

Frameprops are showing me this:


The values are much shorter. That means Decimal() is also totally unnecessary!? And we need to round even more?
Or is FrameProps() rounding them?

[WorBry]

Originally Posted by Iron_Mike

you posted that you also had seek errors w/ option #1, yet when specifically setting seekmode=1 when running the GMSD/SSIM tests you get no errors...

Correct. The seek-test script reported errors (the 3 frame off thing) with ffms2 - Wolfberry's and the version included with VS Fatpack portable. But I see no score inconsistencies running GMSD/SSIM with the same.

[Iron_Mike]

Originally Posted by WorBry

Correct. The seek-test script reported errors (the 3 frame off thing) with ffms2 - Wolfberry's and the version included with VS Fatpack portable. But I see no score inconsistencies running GMSD/SSIM with the same.

are you going through Zopti or using muvsfunc directly ?

[WorBry]

Through Zoptilib

[WorBry]

Originally Posted by WorBry

Well I have now the run tests with x265 CRF28 encoded with custom key frame interval, exactly as per your script:

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

And these are the results I get on serial testing with SSIM:GMSD (Downsample=False):

LWLibavSource:

stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 

ffms2- Wolfberry (default, i.e. seekmode=1)
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786 
stop 435.99385609567906 83.19534706841786

Originally Posted by ChaosKing

I added Decimal() for testing purposes https://github.com/theChaosCoder/zoptilib
The metric values are now much longer, sum should always be the same between runs. Please test.

Retested with Wolfberry's ffms2 (default, seekmode=1) and zoptilib with the Decimal update:

stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637

Consistent results on consecutive testing (still), but slightly different scores to previous results when rounded up to 14 decimal places.....not that it really matters.

[ChaosKing]

Originally Posted by WorBry

Correct. The seek-test script reported errors (the 3 frame off thing) with ffms2 - Wolfberry's and the version included with VS Fatpack portable. But I see no score inconsistencies running GMSD/SSIM with the same.

Maybe you have a consistent inconsistency
I mean your log is always in perfect order... I tested vspipe with 3 frames, and even then sometimes the output is 0 2 1 (before the update)