View Single Post
Old 20th April 2019, 14:33   #1624  |  Link
SmilingWolf
I am maddo saientisto!
 
SmilingWolf's Avatar
 
Join Date: Aug 2018
Posts: 95
Quote:
Originally Posted by benwaggoner View Post
VMAF isn't a still image metric. Has anyone run a correlation for VMAF against subjective testing for still images?
Here you go, based on the TID2013 dataset:
Code:
Actual profile:
Spearman:            | Kendall:
PSNRHA         0.938 | PSNRHA         0.787
PSNRHMA        0.934 | PSNRHMA        0.777
PSNRHVS        0.926 | PSNRHVS        0.766
PSNRHVSM       0.917 | PSNRHVSM       0.749
FSIMc          0.915 | FSIMc          0.742
FSIM           0.911 | FSIM           0.736
WSNR           0.897 | WSNR           0.718
MSSIM          0.887 | MSSIM          0.697
VSNR           0.882 | VSNR           0.690
VMAF_v0.6.1    0.863 | VMAF_v0.6.1    0.675
VMAF_rb_v0.6.3 0.862 | VMAF_rb_v0.6.3 0.674
NQM            0.857 | NQM            0.666
PSNR           0.825 | PSNR           0.624
VIFP           0.815 | VIFP           0.621
PSNRc          0.803 | PSNRc          0.596
SSIM           0.788 | SSIM           0.577

Simple profile:
Spearman:            | Kendall:
PSNRHA         0.953 | PSNRHA         0.818
PSNRHVS        0.951 | PSNRHVS        0.809
FSIM           0.949 | FSIM           0.795
FSIMc          0.947 | FSIMc          0.792
PSNRHVSM       0.938 | PSNRHMA        0.785
PSNRHMA        0.937 | PSNRHVSM       0.780
WSNR           0.933 | WSNR           0.772
PSNR           0.913 | PSNR           0.745
VSNR           0.912 | VSNR           0.731
MSSIM          0.905 | MSSIM          0.720
VIFP           0.897 | VIFP           0.714
VMAF_rb_v0.6.3 0.891 | VMAF_rb_v0.6.3 0.698
VMAF_v0.6.1    0.889 | VMAF_v0.6.1    0.696
PSNRc          0.876 | PSNRc          0.689
NQM            0.875 | NQM            0.681
SSIM           0.837 | SSIM           0.628

Full profile:
Spearman:            | Kendall:
FSIMc          0.851 | FSIMc          0.666
PSNRHA         0.819 | PSNRHA         0.643
PSNRHMA        0.813 | PSNRHMA        0.631
FSIM           0.801 | FSIM           0.629
MSSIM          0.787 | MSSIM          0.607
VMAF_rb_v0.6.3 0.749 | VMAF_rb_v0.6.3 0.564
VMAF_v0.6.1    0.748 | VMAF_v0.6.1    0.563
PSNRc          0.687 | VSNR           0.508
VSNR           0.681 | PSNRHVS        0.507
PSNRHVS        0.654 | PSNRc          0.496
PSNR           0.640 | PSNRHVSM       0.481
SSIM           0.637 | PSNR           0.470
NQM            0.635 | NQM            0.466
PSNRHVSM       0.625 | SSIM           0.463
VIFP           0.608 | VIFP           0.456
WSNR           0.580 | WSNR           0.446
All bitmap images have been converted to raw full range YUV444P with ffmpeg and then measured with the vmafossexec program.
Code:
ffmpeg.exe -i i01_01_1.bmp -vf "scale=flags=accurate_rnd+bitexact+full_chroma_int+full_chroma_inp,format=yuvj444p" i01_01_1.bmp.yuv
vmafossexec.exe yuv444p 512 384 reference_images/i01.bmp.yuv distorted_images/i01_01_1.bmp.yuv model/vmaf_v0.6.1.pkl
vmafossexec.exe yuv444p 512 384 reference_images/i01.bmp.yuv distorted_images/i01_01_1.bmp.yuv model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl --ci
I'm also attaching the raw scores, for completeness sake.

A note on how to read the numbers:
from the paper I get the following: a SROCC of 0.95 is considered excellent, 0.90 is good, and 0.85 is barely acceptable.
Attached Files
File Type: txt VMAF_v0.6.1.txt (29.2 KB, 49 views)
File Type: txt VMAF_rb_v0.6.3.txt (29.2 KB, 41 views)

Last edited by SmilingWolf; 20th April 2019 at 19:09.
SmilingWolf is offline   Reply With Quote