FFMpeg VMAF filter

[Iron_Mike]

Originally Posted by Wolfberry

It works for me using a relative path.
I also tried libvmaf=model_path=vmaf_b_v0.6.3.pkl:enable_conf_interval=1 and that also works.
Something must be wrong on your side, but still good to know that the vapoursynth filter works.

did you test the relative path on Windows ?

I cannot get the .pkl path to work on your ffmpeg Win build... I tried relative path (file in same folder as ffmpeg.exe), full Windows path or Unix style path... it throws the

Caught VmafException: Error loading model (.pkl):

error...

[Wolfberry]

ffmpeg -i 1.mkv -i 1.mkv -lavfi libvmaf=model_path=vmaf_v0.6.1.pkl:log_path=vmaf.log:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -f null -

vmaf log / ffmpeg log I'm using Windows 10 18346.1 Education

[excellentswordfight]

Originally Posted by Iron_Mike

did you test the relative path on Windows ?

I cannot get the .pkl path to work on your ffmpeg Win build... I tried relative path (file in same folder as ffmpeg.exe), full Windows path or Unix style path... it throws the error...

Same issue here, tried all the variants i could think of and tried both 0.6.1 and 0.6.3

Caught VmafException: Error loading model (.pkl): Trouble reading the file:vmaf_v0.6.3.pkl

or

Input model at vmaf_v0.6.1.pkl cannot be read successfully.
Caught VmafException: Error loading model (.pkl): Unknown token

[Iron_Mike]

okay, made some progress...

if you call "ffmpeg" from CLI because you're either already in that folder or location (of that ffmpeg version) is on Windows PATH, then the relative path to VMAF will work !

but if you call ffmpeg with a fully qualified path (e.g. "d:/path/to/ffmpeg.exe" -i test.mkv.... etc ) then the relative path to the vmaf.pkl will not work... and for whatever reason I cannot seem to get fully qualified paths to the .pkl files working...

[Wolfberry]

Yes, exactly that. Future updates will be in my signature, update frequency will probably be every other day.

[poisondeathray]

Thanks Wolfberry, (and for vpy support!)

For people having syntax issues - For Windoze, the issue with ffmpeg and filters is escape characters . To enter full paths, for every colon , you need to enter a preceding backslash. This is true for all filters syntax in windows ffmpeg with full paths

eg.
(I marked in red in this example)

-lavfi "libvmaf=model_path='C\:/Program Files (x86)/VapourSynth/plugins64/VMAF-r3/plugins64/model/vmaf_v0.6.1.pkl':log_path='F\:/TEST/vmaf.log':log_fmt=json:psnr=1:ssim=1:ms_ssim=1"

[Iron_Mike]

Originally Posted by poisondeathray

Thanks Wolfberry, (and for vpy support!)

For people having syntax issues - For Windoze, the issue with ffmpeg and filters is escape characters . To enter full paths, for every colon , you need to enter a preceding backslash. This is true for all filters syntax in windows ffmpeg with full paths

eg.
(I marked in red in this example)

-lavfi "libvmaf=model_path='C\:/Program Files (x86)/VapourSynth/plugins64/VMAF-r3/plugins64/model/vmaf_v0.6.1.pkl':log_path='F\:/TEST/vmaf.log':log_fmt=json:psnr=1:ssim=1:ms_ssim=1"

Thanks for that, I tried that before (same escape syntax as specifying subtitle files) but it didn't work...

but got it to work now by using your exact syntax of wrapping the full VMAF filter command in double quotes, then wrapping the .pkl path in single quotes (that was t6he missing part), and then escaping the colon...

Thanks !

[Iron_Mike]

follow up question in regards to using VMAF:

I understand that one needs to upscale a video to the source resolution in order to compare via VMAF, but what about bit depth ?

so if I compare a 16-bit EXR sequence (full RGB) with a 12-bit 444 encode (yuv444p12le) - both in same resolution - do I need to adjust anything for the VMAF comparison to be as accurate as possible ?

read the VMAF FAQs but it does not mention bit depth differences...

[Beelzebubu]

If the encoded video and the source video have a different bitdepth, then yes, you need to choose what to measure it at. I would recommend to use the evaluation bitdepth used by the display device, and make sure to use a model tuned to that bitdepth (and resolution) also.

[Iron_Mike]

Originally Posted by Beelzebubu

If the encoded video and the source video have a different bitdepth, then yes, you need to choose what to measure it at. I would recommend to use the evaluation bitdepth used by the display device, and make sure to use a model tuned to that bitdepth (and resolution) also.

could you provide an example of the ffmpeg command ?

I've tried passing in different pixel formats (for both inputs), VMAF score never changed...

both vids are HD res, so I used model 0.6.1...

lmk what parameters to change and I'll run tests...

Thanks.

[foxyshadis]

Originally Posted by Iron_Mike

follow up question in regards to using VMAF:

I understand that one needs to upscale a video to the source resolution in order to compare via VMAF, but what about bit depth ?

so if I compare a 16-bit EXR sequence (full RGB) with a 12-bit 444 encode (yuv444p12le) - both in same resolution - do I need to adjust anything for the VMAF comparison to be as accurate as possible ?

read the VMAF FAQs but it does not mention bit depth differences...

VMAF only goes up to 10-bit, so ffmpeg's either converting both to 10-bit or to 8-bit before passing them on to VMAF.

[rco133]

Hi.

I think I need some help with libvmaf.

No matter what I do, after ffmpeg has scanned the files it ends up with VMAF nan

The version of ffmpeg used is the most recent downloaded from Wolfberry.

My command line is:

ffmpeg -i encoded.mkv -i lossless.mkv -lavfi libvmaf=vmaf_4k_v0.6.1.pkl -f null -

The ouput of ffmpeg is:

Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x1608 [SAR 1:1 DAR 160:67], 23.98 fps, 23.98 tbr, 1200k tbn, 23.98 tbc

Stream #1:0: Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x1608, 23.98 tbr, 1200k tbn, 23.98 tbc

Stream mapping:
Stream #0:0 (hevc) -> libvmaf:main
Stream #1:0 (hevc) -> libvmaf:reference
libvmaf -> Stream #0:0 (wrapped_avframe)

frame= 0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A speed=N/A Start calculating VMAF score...

Output #0, null, to 'pipe:':
Metadata:
encoder : Lavf58.31.104
Stream #0:0: Video: wrapped_avframe, yuv420p10le, 3840x1608 [SAR 1:1 DAR 160:67], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn, 23.98 tbc (default)
Metadata:
encoder : Lavc58.55.101 wrapped_avframe

And then it takes quite a while before the ouput comes. The files have a bit more than 6000 frames.

After scanning all the frames the output I get is this:

frame= 6151 fps=8.3 q=-0.0 Lsize=N/A time=00:04:16.54 bitrate=N/A speed=0.348x
video:3220kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Exec FPS: 8.323916
VMAF score = nan
[libvmaf @ 00000284353da5c0] VMAF score: nan

I have managed to get vmafossexec.exe to output some numbers, but it only supports raw yuv files, and I would like to doublecheck the numbers as the PSNR and SSIM numbers from vmafossexec.exe, doesn't match what I have previously gotten from ffmpeg.

Any help will be most appreciated.

rco133

[benwaggoner]

Originally Posted by foxyshadis

VMAF only goes up to 10-bit, so ffmpeg's either converting both to 10-bit or to 8-bit before passing them on to VMAF.

If you are comparing encodes made from a source that gets color space converted, you need to compare with the actual frames that the encoder is taking as input. So the scaled 8/10-bit Y'CbCr 4:2:0.

If you can do a lossless encode using the same tool, that would work. like if using ffmpeg, pipe out to .y4m.

[sashaItats]

A second note on the filter usage:
- as in all my last filters, I make use of random selections tha plasma applied is randomly selected, the pattern also
- hence, if you do not like one outcome just relaunch the filter and you will get a different one, even with the same parameters

[orion44]

Originally Posted by Wolfberry

ffmpeg-4.2-N-93255-g4571c7c05d built with gcc 8.3.1 (G41) 20190222

The vapoursynth dependencies is now included.

The build can also be found in my signature, future updates will be there.

configuration: --disable-autodetect --disable-debug --disable-w32threads --enable-amf --enable-avisynth --enable-bzlib --enable-cuda --enable-d3d11va --enable-ffnvcodec --enable-iconv --enable-libaom --enable-libass
--enable-libbluray --enable-libdav1d --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-liblensfun --enable-libmfx --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus 
--enable-librubberband --enable-libsoxr --enable-libsrt --enable-libssh --enable-libtesseract --enable-libvidstab --enable-libvmaf --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 
--enable-libzimg --enable-lzma --enable-mbedtls --enable-nvdec --enable-nvenc--enable-opencl --enable-pthreads --enable-sdl2 --enable-vapoursynth --enable-zlib --enable-gpl --enable-version3

libavutil      56. 26.100 / 56. 26.100
libavcodec     58. 47.102 / 58. 47.102
libavformat    58. 26.101 / 58. 26.101
libavdevice    58.  6.101 / 58.  6.101
libavfilter     7. 48.100 /  7. 48.100
libswscale      5.  4.100 /  5.  4.100
libswresample   3.  4.100 /  3.  4.100
libpostproc    55.  4.100 / 55.  4.100

Comparison with Zeranoe builds:

w/ libfribidi liblensfun librubberband libsrt libssh libtesseract libvmaf mbedtls opencl vapoursynth
w/o gmp gnutls libmp3lame libopencore-amrnb libopencore-amrwb libshine libsnappy libspeex libtheora libtwolame libvorbis libvo-amrwbenc libwavpack libxvid

The link for FFmpeg is dead. I really want to try out the VMAF filter.

[Vincent Vega]

is there a way to encode to say x264(software) or h264(nvidia hardware) and to simultanuously pipe/tee input and output frames to libvmaf for comparison?
i.e to do encode and vmaf calculation with single ffmpeg command?

[benwaggoner]

Originally Posted by Vincent Vega

is there a way to encode to say x264(software) or h264(nvidia hardware) and to simultanuously pipe/tee input and output frames to libvmaf for comparison?
i.e to do encode and vmaf calculation with single ffmpeg command?

That should be feasible, and it's a lot faster to calculate the base metrics while the source and output frames are both in RAM together. So many erroneous VMAF results come from not quite getting frames to be apples-to-apples when decoding the source again, using suboptimal scaling, etcetera. Comparing the post-processed 4:2:0 frame that's going into the encoder, and the reconstructed frame that has been encoding, is a lot harder to mess up.

There's no reason the ML calculations need to happen while encoding; generating the base metrics is the slow part, and the VMAF calculations can be done later.

I like that approach as it allows for running different VMAF models without having to redo those calculations.