Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 27th June 2019, 03:16   #1  |  Link
heisenberg
Registered User
 
Join Date: May 2019
Posts: 7
Poor PSNR & VMAF scores

Long winded, sorry, but want to make sure all details are given.

I ripped a BD to uncompressed yuv422. I chose that format because it was easiest to work with in my NLE. After editing, I saved the result also as uncompressed yuv422 in MXF format. I extracted a 15m clip of the darkest scene to avi container using ffmpeg with -c:v copy. Here's what ffmpeg says about the clip:

Stream #1:0: Video: rawvideo (Y42B / 0x42323459), yuv422p, 1920x816, 601047 kb/s, SAR 1:1 DAR 40:17, 23.98 fps, 23.98 tbr, 23.98 tbn, 23.98 tbc

I did many different test encodes using handbrake on mac. Here's what ffmpeg says about the encodes:

Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x816 [SAR 1:1 DAR 40:17], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn, 23.98 tbc (default)

I then used ffmpeg (again on mac) to do a libvmaf analysis, using the default model. The results are quite poor. Some selected samples:

Code:
	psnr	vmaf	data rate
crf12	44.283	75.858	30.5
crf17	42.622	74.751	11.4
crf20	41.723	73.444	5.2
If you are knowledgable about vmaf, you know that the training model is limited to crf22. not exceptional. so at least i expect to be at 96+. The peak ranges for crf17 are there. For crf20 it's closer to 92. Here's where things get interesting. Here's the pattern I see on a frame-by-frame basis:

Code:
frame	adm2	motion2	vmaf-psnr	vmaf
0	0.99415	0	51.48603	96.04917
1	0.98217	0	46.51105	92.86528
2	0.86527	0	25.305		41.55384
3	0.88616	5.47273	24.02293	50.6055
4	0.98737	4.50818	47.7387		100
5	0.98205	0	46.20227	92.81662
6	0.8591	0	25.63469	43.84714
7	0.98358	4.90082	46.8299		99.65214
8	0.98082	0	46.06436	92.55462
9	0.81607	0	25.2691		31.84736
10	0.8323	5.02721	26.03205	39.83634
11	0.97943	4.01224	45.83572	97.3602
12	0.98223	0	46.05179	92.76931
13	0.85919	0	26.81215	39.87247
Note that the ffmpeg psnr filter calculates values that vary with this same pattern. (not shown above). So, I don't think this is some anomaly with the libvmaf psnr.

The frames are definitely not a stuttering motion. It's a live-action film, and not telecined etc.

Am I doing something wrong with the encoding? I'm using

Code:
subme=10:deblock=-1,-1:me=umh:merange=32:mbtree=0:dct-decimate=0:fast-pskip=0:aq-mode=2:
psy-rd=1,0.15:vbv-maxrate=62500:level=4.1:ref=5:bframes=16:b-adapt=2:direct=auto:analyse=all:trellis=2:
vbv-bufsize=78125:rc-lookahead=60
The deblock and psy-rd are from "film tune". I have tested other values as well but these don't generally change the vmaf/psnr/motion2 scores.

I've tried level 5.1 in case i'm being buffer limited but i think crf17 is making that have no effect.

I've tested up to crf 9 and I only get up to about 76 vmaf, and with the same motion2/psnr pattern.

Lastly, if I reduce the ref.avi to yuv420p, i get the exactly same frame-by-frame scores for all metrics. This is what you'd expect since vmaf only considers luma.

I've got myself fairly well convinced this is an encoding problem, not a vmaf analysis problem. Any ideas?!?

Last edited by heisenberg; 27th June 2019 at 13:48.
heisenberg is offline   Reply With Quote
Old 27th June 2019, 03:47   #2  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
It suggests alignment issue ( mismatches - not comparing same frame) . FFmpeg isn't always frame accurate with seeks , and VMAF has a temporal component .

Also possibility of some issue with your "extraction" . How did you do this ?

Check the timecodes of source and encode to see if they match

You can decode the encode to some lossless I frame format, and compare that against the reference - that will help if it's a ffmpeg seek/decoding issue from your encoding

Or use vapoursynth, with indexed frame accurate source filter to force accurate seeking
poisondeathray is offline   Reply With Quote
Old 27th June 2019, 13:36   #3  |  Link
heisenberg
Registered User
 
Join Date: May 2019
Posts: 7
Quote:
It suggests alignment issue ( mismatches - not comparing same frame).
I thought that initially, because opening the avi in vlc shows that the first frame repeats! (While first frame of the encode does not.) But opening it any other way (eg mpv) does not exhibit that problem. How can I investigate this further?

I read that ffmpeg is nowadays frame-accurate with extraction, unless you explicitly tell it not to be, but regardless I don't think that is the issue.

I used ffmpeg to go from the source m2ts to a different uncompressed container. I have to check later what container (not sure if avi or other). I did some (lots) of NLE and then saved as uncompressed yuv422p MXF. I used ffmpeg to copy to avi. I then used ffmpeg to extract the 15 minutes. I just did a simple ffmpeg -ss HH:MM:SS -t MM:SS input.avi -c:v copy output.avi. Visually, the extraction was frame accurate. I'll verify that the duration is exactly what I intended.

Then I encoded only that 15 min avi. And the libvmaf analysis was across the entirety of the 2 files (ref avi and encoded mkv). The 2 files have the same frame count, verified by ffmpeg copy to a null output.

So if there is a seek issue, it would have affected that 15 min ref avi, and been propagated to the encodes.

Maybe it's a timecode issue in the avi? mpv is ignoring it, but vlc is honoring it and that's why the first frame repeats? It would mean that handbrake also ignores the timecode issue, and doesn't propagate it to the encoded mkv. Then, in analysis, ffmpeg isn't actually progressing frame-by-frame but is rather doing something with the funky timecode? hmm.

Quote:
Check the timecodes of source and encode to see if they match
ok well i'll start here.

Last edited by heisenberg; 27th June 2019 at 13:38.
heisenberg is offline   Reply With Quote
Old 27th June 2019, 14:27   #4  |  Link
richardpl
Registered User
 
Join Date: Jan 2012
Posts: 272
You could use setpts='N*TB' filter for each input prior to calling psnr filter. This is needed if timestamp do not match between containers.
richardpl is offline   Reply With Quote
Old 27th June 2019, 17:17   #5  |  Link
heisenberg
Registered User
 
Join Date: May 2019
Posts: 7
libvmaf analysis always gives me this output at the end:

Code:
ffmpeg -i encode.mkv -i ref.avi -lavfi libvmaf=... -f null -
...
[null @ 0x7ffc02801200] Application provided invalid, non monotonically
increasing dts to muxer in stream 0: 21674 >= 21674
I was just ignoring it, thinking it is a normal message from forcing the null output. But is this actually a tell-tale for the temporal problem? There are 21676 frames (0-21675).
heisenberg is offline   Reply With Quote
Old 27th June 2019, 18:04   #6  |  Link
heisenberg
Registered User
 
Join Date: May 2019
Posts: 7
Ah yes, it must be the timecode.

Code:
ffmpeg -i ref.avi -c:v copy ref-pts.avi
ffmpeg -i test-crf20.mkv -c:v rawvideo crf20-pts.avi
ffmpeg -i crf20pts.avi -i ref-pts.avi -lavfi libvmaf -f null -
...
VMAF score = 93.281505 # previously 73.44370
I haven't inspected the frame-by-frame yet but clearly it's cleaned up, for the avg to be up where it should be.

Any tips on how I can inspect the timecode values and thereby see where I went wrong in the process?

Last edited by heisenberg; 29th June 2019 at 02:56.
heisenberg is offline   Reply With Quote
Old 28th June 2019, 08:41   #7  |  Link
richardpl
Registered User
 
Join Date: Jan 2012
Posts: 272
Its timestamps issue, timecode is something else.

You can inspect timestamps (pts) with showinfo filter.
richardpl is offline   Reply With Quote
Old 28th June 2019, 15:14   #8  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Timestamp is the proper name in FFMpeg. PTS (presentation timestamp) used to be called "display timecodes" in other programs

FFProbe should be able to print a list of PTS, DTS , or ffmsindex from ffms2
poisondeathray is offline   Reply With Quote
Old 29th June 2019, 03:10   #9  |  Link
heisenberg
Registered User
 
Join Date: May 2019
Posts: 7
Using ffmpeg -vf showinfo, for my rawvideo avi (created by ffmpeg), I see:

Code:
[Parsed_showinfo_0 @ 0x7fcfd0400100] config in time_base: 1001/24000, frame_rate: 24000/1001
...
[Parsed_showinfo_0 @ 0x7fcfd0400100] n:   1 pts:      1 pts_time:0.0417083
[Parsed_showinfo_0 @ 0x7fcfd0400100] n:   2 pts:      2 pts_time:0.0834167
Note that the pts_time is exactly correct, ie each frame is shown at 1001/24000 sec intervals.

For a rawvideo mkv (avi copied into a mkv container by ffmpeg), as well as the x264 encoded mkv, I see:

Code:
[Parsed_showinfo_0 @ 0x7ff503f03780] config in time_base: 1/1000, frame_rate: 24000/1001
[Parsed_showinfo_0 @ 0x7ff503f03780] n:   1 pts:     42 pts_time:0.042
[Parsed_showinfo_0 @ 0x7ff503f03780] n:   2 pts:     83 pts_time:0.083
Note that there is a small error in the time due to the imprecise time_base.

I guess even all other things being perfect this will result in a wrong vmaf analysis for very long videos. Even for short videos, I guess the first error occurs at frame 1 and then the next error at frame 24000.

But when comparing avi-to-avi, the timestamps both match, so no error. and presumably, mkv-to-mkv would also have no such error.

When watching either of the rawvideo files (avi or mkv) with vlc, both repeat frame 0. The encoded mkv does not. I guess this is a vlc bug related to rawvideo and not the timestamps.

The other curiosity is, what happened for time 0.000? is that n=0 and it just isn't shown by ffmpeg? Seems odd it would skip that frame.

Last edited by heisenberg; 29th June 2019 at 03:20.
heisenberg is offline   Reply With Quote
Old 29th June 2019, 10:21   #10  |  Link
richardpl
Registered User
 
Join Date: Jan 2012
Posts: 272
Can't tell without access to actual file.
richardpl is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 01:17.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.