Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > New and alternative video codecs

Reply
 
Thread Tools Search this Thread Display Modes
Old 10th November 2024, 09:58   #1  |  Link
birdie
Artem S. Tashkinov
 
birdie's Avatar
 
Join Date: Dec 2006
Posts: 394
Why metrics like PSNR, SSIM, VMAF suck while exploring possible alternatives

A nice article on streaming media.
birdie is offline   Reply With Quote
Old 10th November 2024, 15:17   #2  |  Link
Z2697
Registered User
 
Join Date: Aug 2024
Posts: 228
I don't really like the "FGS and AI" themed topic though.
Z2697 is offline   Reply With Quote
Old 10th November 2024, 16:17   #3  |  Link
Z2697
Registered User
 
Join Date: Aug 2024
Posts: 228
I don't think they (terribly) suck, they are "misused", if you know they suck at something then avoid it... for example SSIM fails to "detect" blur, or the non-neg VMAF models give much higher score to sharpened video.
Which might result in you should only trust the metric scores published online or you tested yourself when you can rule out those uncertainties. What a harsh world.
Current metrics, they have shortcomings. New metrics, new shortcomings.
Z2697 is offline   Reply With Quote
Old 11th November 2024, 02:25   #4  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,904
The least-bad metric I've used is the ITU p1204, which has a mode that combines both full reference and bitstream analysis. Of course it uses machine learning, so having a good ground truth data set that aligns with your scenarios really helps. And which is also very expensive and labor intensive to make.

It's important to distinguish between a distortion metric (like PSNR, SSIM, and VMAF) with a quality metric (like p1204), of course. A distortion metric will rate a very accurate encode of a horrible looking source higher than a still pretty accurate encode of a great source. A quality metric will measure how good it looks inclusive of both source and encoding artifacts.

Distortion metrics are great for tuning an encoder. Quality metrics are great for estimating how good a viewer will think the content looks. They're not so different that one can't be used for the other, but it's best to use the right kind of metric for what you're trying to accomplish.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Reply

Tags
psnr, quality analyse, ssim, vmaf

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 21:25.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.