Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 7th November 2018, 11:41   #1  |  Link
HolyWu
Registered User
 
Join Date: Aug 2006
Location: Taiwan
Posts: 392
VMAF - Video Multi-Method Assessment Fusion

https://github.com/HomeOfVapourSynth...ourSynth-VMAF/
HolyWu is offline  
Old 7th November 2018, 16:11   #2  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Thanks
poisondeathray is offline  
Old 7th November 2018, 16:37   #3  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
I'm confused, what is this filter supposed to do?
lansing is offline  
Old 7th November 2018, 16:52   #4  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
It calculates a score.... You have 2 clips, the score tells you by how much the the seconds clips differs from the first one. The VMAF algo tries take the human perception into account. https://en.wikipedia.org/wiki/Video_...essment_Fusion
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database
ChaosKing is offline  
Old 7th November 2018, 20:07   #5  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
Nice thanks!
__________________
Hybrid here in the forum, homepage
Selur is offline  
Old 8th November 2018, 11:11   #6  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
Also available via vsrepo now
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database
ChaosKing is offline  
Old 9th November 2018, 02:56   #7  |  Link
ifb
Registered User
 
Join Date: Dec 2009
Posts: 72
Very timely for me. Thanks.

I've had mixed results getting yuv422p10le to work correctly with vmafossexec and/or ffmpeg builds with libvmaf. That's probably my fault for trying on Windows, but I was too lazy to try on a Linux VM at the time.
ifb is offline  
Old 13th November 2018, 18:23   #8  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Request: is it possible to print out the aggregate psnr, ssim, ms_ssim scores ? (Currently it's only aggregate vmaf)
poisondeathray is offline  
Old 14th November 2018, 12:21   #9  |  Link
HolyWu
Registered User
 
Join Date: Aug 2006
Location: Taiwan
Posts: 392
Update r2.
  • Scale 10-bit pixel values to 8-bit range for correct score calculation.
  • Use stricter linear frame request since VMAF score will change if frame order is different.
  • Report aggregate PSNR, SSIM, and MS-SSIM scores in addition to VMAF score.
HolyWu is offline  
Old 1st March 2019, 05:51   #10  |  Link
Iron_Mike
Registered User
 
Join Date: Jul 2010
Posts: 132
Quote:
Originally Posted by HolyWu View Post
Update r2.
  • Scale 10-bit pixel values to 8-bit range for correct score calculation.
  • Use stricter linear frame request since VMAF score will change if frame order is different.
  • Report aggregate PSNR, SSIM, and MS-SSIM scores in addition to VMAF score.
could you elaborate on the first two points of this update ?

does that mean we have to scale both inputs to 8bit before calculating VMAF, or does VMAF do it automatically ?

I'm comparing a 16bit exr sequence to itself (as control) and I get a 98.2 VMAF score... I know that the FAQ states that this is normal, just making sure I do this correctly... most of the examples here use 8bit source/ref footage...

When I compare a 12bit 444 (yuv444p12le) CRF 10 x265 encode to the 16 bit exr ref footage I get a 96.4 VMAF score... little bit low considering the tests that WorBry has done...

Also, how do I use a "stricter linear frame request" ?

Thanks.

Last edited by Iron_Mike; 1st March 2019 at 05:55.
Iron_Mike is offline  
Old 1st March 2019, 15:19   #11  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
Quote:
Originally Posted by Iron_Mike View Post
could you elaborate on the first two points of this update ?

does that mean we have to scale both inputs to 8bit before calculating VMAF, or does VMAF do it automatically ?

.....Also, how do I use a "stricter linear frame request" ?
These are internal improvements that were made in update r2 - you don't need to do anything.

Quote:
Originally Posted by Iron_Mike View Post
When I compare a 12bit 444 (yuv444p12le) CRF 10 x265 encode to the 16 bit exr ref footage I get a 96.4 VMAF score... little bit low considering the tests that WorBry has done...
Bear in mind though that all of those tests were done on a single source. Could be any number of factors weighing in there. What scores to you get if you get if you test the ref and x265 clips against themselves ?
__________________
Nostalgia's not what it used to be

Last edited by WorBry; 1st March 2019 at 15:57.
WorBry is offline  
Old 1st March 2019, 22:42   #12  |  Link
Iron_Mike
Registered User
 
Join Date: Jul 2010
Posts: 132
Quote:
Originally Posted by WorBry View Post
Bear in mind though that all of those tests were done on a single source. Could be any number of factors weighing in there. What scores to you get if you get if you test the ref and x265 clips against themselves ?
I ran all test w/ the ffmpeg libvmaf filter, but I assume since it uses the exact same VMAF models the result would be the same...

16bit EXR ref clip is 400 frames - control test to itself via 0.6.1 results in 98.2

x265 12bit 444 encode (from EXR), control tested to itself via 0.6.1 results in 98.08

x265 12bit 444 encode (from EXR) tested against it's source (16 bit EXR) via 0.6.1 results in 96.4


interesting that the clip you tested had 399 out of 400 frames a perfect 100 in the control test...
Iron_Mike is offline  
Old 1st March 2019, 23:09   #13  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
Quote:
Originally Posted by Iron_Mike View Post
..interesting that the clip you tested had 399 out of 400 frames a perfect 100 in the control test...
For that particular source, yes...actually it was 499 out of 500 frames that scored 100 - it was just that first frame with the motion2 score of 0 that skewed the aggregate score. But I've yet to test other sources.

Go through the list of per-frame VMAF scores from your 'self' tests and you'll be able to identify which frames are skewing the aggregate score.
__________________
Nostalgia's not what it used to be
WorBry is offline  
Old 1st March 2019, 16:43   #14  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by Iron_Mike View Post

When I compare a 12bit 444 (yuv444p12le) CRF 10 x265 encode to the 16 bit exr ref footage I get a 96.4 VMAF score... little bit low considering the tests that WorBry has done...

I'm not sure how valid that test would be. EXR is usually sRGB linear , and 16bit half float .

For any metric you usually need a common ground to compare. This means same pixel format (same colorspace, same bit depth, same chroma subsampling) . Otherwise you introduce other variables that are not controlled for. e.g. if one run uses one algorithm to scale (e.g. bicubic vs. bilinear, vs...) , or another dithers down using one algorithm, but another does not... or if you convert to RGB using different matrix, etc... there are many factors that invalidate your testing
poisondeathray is offline  
Old 1st March 2019, 22:12   #15  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
Not to mention the potential for frame shifts/misalignment when using different decoders for the reference and test clips, although the filter will report an error if the number of frames is different.

Also needs to be appreciated that the VMAF models are 'trained' for predicting perceptual quality at streaming bitrates primarily. Came across this quote from Netflix:

Quote:
"VMAF has been trained using encodes spanning from CRF 22 @ 1080 (highest quality) to CRF 28 @ 240 (lowest quality). The former is mapped to score 100 and the latter is mapped to score 20. Anything in between is mapped in the middle (for example, SD encode at 480 is typically mapped to 40 ~ 70)."
https://streaminglearningcenter.com/...flix-vmaf.html

I would assume a similar focus was applied in training the 4K model.

So at CRF 10 you are well into uncharted territory.

Personally, I'd be more inclined to look at other metrics available for VapourSynth that are (maybe) better attuned for VQA in the visually lossless domain - GMSD, MDSI and yes, SSIM....Butteraugli, possibly. My own journey of discovery in that vein continues:

https://forum.doom9.org/showthread.php?t=176101
__________________
Nostalgia's not what it used to be

Last edited by WorBry; 1st March 2019 at 22:19.
WorBry is offline  
Old 1st March 2019, 22:59   #16  |  Link
Iron_Mike
Registered User
 
Join Date: Jul 2010
Posts: 132
Quote:
Originally Posted by WorBry View Post
Not to mention the potential for frame shifts/misalignment when using different decoders for the reference and test clips, although the filter will report an error if the number of frames is different.
not sure what "different encoders" here relates to ?

ffmpeg reads an EXR frame and then reads a frame from the mp4 encode (that was done from the EXR via ffmpeg) and then passes the decoded frames to VMAF for comparison...

is there a setting to avoid frame shifts/misalignment ?


Quote:
Originally Posted by WorBry View Post
So at CRF 10 you are well into uncharted territory.
you yourself validated the test clip up to CRF 0 and results on the charts make sense... your encodes went up to VMAF 100... not sure I understand your concern for CRF 10 ?

Thanks

Last edited by Iron_Mike; 1st March 2019 at 23:07.
Iron_Mike is offline  
Old 1st March 2019, 22:51   #17  |  Link
Iron_Mike
Registered User
 
Join Date: Jul 2010
Posts: 132
Quote:
Originally Posted by poisondeathray View Post
I'm not sure how valid that test would be. EXR is usually sRGB linear , and 16bit half float .

For any metric you usually need a common ground to compare. This means same pixel format (same colorspace, same bit depth, same chroma subsampling) . Otherwise you introduce other variables that are not controlled for. e.g. if one run uses one algorithm to scale (e.g. bicubic vs. bilinear, vs...) , or another dithers down using one algorithm, but another does not... or if you convert to RGB using different matrix, etc... there are many factors that invalidate your testing
EXR contains whatever you put into it (has nothing to do with sRGB) - there are no assumptions here: this is the master of the movie in HD in lossless 16 bit EXR, I took 400 frames as a test sequence

for streaming the movie was encoded via ffmpeg and x265 from EXR (rgb48le) to x265 12bit 444 (yuv444p12le) - final result looks very good, we're using VMAF to compare various encodes (presets/CRF/etc) against each other... exact same as NF does it w/ VMAF... the master one delivers to NF is obviosly also not 8bit 420, they encode from that (high quality) master for NF streaming...

so the "common ground" you state is the same movie in the same resolution, which is the only thing that NF states in their VMAF instructions...

the whole reason for comparison is different output bit depth w/ different output chroma subsampling, on top of different encoding settings, so I do not understand your point...

Last edited by Iron_Mike; 1st March 2019 at 22:53.
Iron_Mike is offline  
Old 14th November 2018, 15:09   #18  |  Link
edcrfv94
Registered User
 
Join Date: Apr 2015
Posts: 84
Is it possible to registered to frame props like mvsfunc PlaneStatistics?
edcrfv94 is offline  
Old 14th November 2018, 15:51   #19  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
Quote:
Originally Posted by edcrfv94 View Post
Is it possible to registered to frame props like mvsfunc PlaneStatistics?
This would be awesome.
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database
ChaosKing is offline  
Old 15th November 2018, 03:43   #20  |  Link
HolyWu
Registered User
 
Join Date: Aug 2006
Location: Taiwan
Posts: 392
Quote:
Originally Posted by edcrfv94 View Post
Is it possible to registered to frame props like mvsfunc PlaneStatistics?
Not possible with the exposed API of libvmaf. Only when all frames are deliverd and processed does the library print the aggregate scores and optionally write the log file. There is no way to access the library's internal data to get the per-frame score and add that to the frame property.

Last edited by HolyWu; 15th November 2018 at 03:57.
HolyWu is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:05.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.