JVQA and Shifted Gradient Similarity index development thread

[Hyral]

Hello fellow encoding professionals,

Please allow me to present my recent academic research and development on SSIM-based VQA metrics. I have developed the Video Quality Assessment in Java - jVQA tool, inspired by Moscow State University's Video Quality Measurement Tool, and the Shifted Gradient Similarity (SG-Sim) index, based on the Fast SSIM index by Chen & Bovik and on considerations by Rouse & Hemami. I hope my work comes as a contribution to the video encoding community.

JVQA is available at http://sourceforge.net/projects/jvqa/ under the GPL license. Its primary goals are to allow quality assessment of HD content whilst VQMT requires a $1000 license (and jVQA actually supports 4K as well), and to be able to customize the SSIM algorithm with several interchangeable component techniques in order to identify those that produce the highest DMOS correlation (i.e., subjective quality) as well as the most computationally efficient. It is designed for reuse and extension, with object-oriented design patterns.

A lot of work remains to be done in code organization, utility, performance and user interface, but it is very usable and provides basic command-line and graphical user interfaces. JVQA can generate quality maps, samples of which are available at the project page at SourceForge, but for the time being this is only possible with test classes within an IDE (no interface commands are yet available for exporting quality maps).

JVQA was developed in Java primarily because I was severely time-constrained and I'm most familiar and efficient with Java, but I also hoped it could be convenient as a multi-platform tool. However, dependency on FFmpeg and AviSynth restrict that advantage.

JVQA allowed me to develop the SG-Sim index which works the mathematical pitfalls of classic SSIM metrics to improve their DMOS correlation. The results are to be published at the IEEE International Conference on Visual Communications and Image Processing (VCIP2015, http://vcip2015.org) this december at Singapore, under the title "Perceptual video quality assessment for adaptive streaming encoding". My experiments with the LIVE Mobile VQA dataset place SG-Sim as perceptually more accurate than SSIM, 3-SSIM, 4-SSIM, Fast SSIM and GMSD.

Also, yet-unpublished results indicate that SG-Sim highly correlates with the Psy-RDO of x264, accurately ranking Psy-RDO > SSIM-RDO > PSNR-RDO. Today, Psy-RDO is said to be subjectively better than the other RD modes with little or inconclusive academic backing; SG-Sim now provides an objective measure of the higher quality of Psy-RDO, based on subjective testing by means of DMOS. This could be an asset to x264, x265 and probably any other encoder. These formal results should be published within the next semester.



The following is the README file for jVQA:

jVQA: VIDEO QUALITY ASSESSMENT IN JAVA
======================================
Version: 1.0-beta3 28-aug-2015.

Based on the Fast Structural Similarity index proposed by Chen and Bovik:
http://live.ece.utexas.edu/publicati..._rtip_2011.pdf

Implements the original, variance-based SSIM, Multi-Scale SSIM, Fast SSIM, 2, 3
and 4-Component Weighted SSIM, Gradient Magnitude Similarity Deviation, and
our original Shifted Gradient Similarity (SG-Sim) index. Indexes may be
customized by selecting image structure statistic (variance, gradient, shifted
gradient, 2/3/4-component gradient of variance), structure scale (full,
downsampled to 256, 5-scaled or 4-scaled), pooling window type (global/
unfiltered, 3-standard-deviations Gaussian, 2-standard-deviations (optimized)
Gaussian, Box or downsampling Box) and size, index stabilization (logical or
by constants) and luminance index (global/unfiltered, 3-standard-deviations
Gaussian, 2-standard- deviations (optimized) Gaussian, Box, downsampling Box or
no luminance index).

Ultra High Definition content supported, requiring sufficient memory to be
allocated to the Java Virtual Machine instance running jVQA.

Built upon JavaCPP (http://github.com/bytedeco/javacpp), FFmpeg
(http://www.ffmpeg.org) and Java Native Access for Avisynth - jnavi
(https//sourceforge.net/projects/jnavi).

"JavaCPP provides efficient access to native C++ inside Java. Under the hood,
it uses JNI, so it works with all implementations of Java SE."
- http://github.com/bytedeco/javacpp

"FFmpeg is the leading multimedia framework, able to decode, encode, transcode,
mux, demux, stream, filter and play pretty much anything that humans and
machines have created. It supports the most obscure ancient formats up to the
cutting edge. No matter if they were designed by some standards committee, the
community or a corporation."
- http://www.ffmpeg.org

"AviSynth is a powerful tool for video post-production. It provides ways of
editing and processing videos. AviSynth works as a frameserver, providing
instant editing without the need for temporary files." - http://avisynth.org

"JNA provides Java programs easy access to native shared libraries without
writing anything but Java code - no JNI or native code is required. This
functionality is comparable to Windows' Platform/Invoke and Python's ctypes."
- https://github.com/twall/jna

Source code organized with Apache Maven (http://maven.apache.org).


USER GUIDE
==========

Operating system: While jVQA itself is multi-platform through Java Virtual
Machine, it depends on native builds of FFmpeg. Use the appropriate build for
your system or build it yourself with Maven.

32-bit vs. 64 bit: The included FFmpeg build must also be appropriate for the
Java Runtime Environment upon which jVQA is to run. A 32-bit FFmpeg compilation
will not run on a 64-bit JRE, and vice-versa.

AviSynth support: The AviSynth frameserver must be installed separately. The
standard distribution is 32-bit, which will require the 32-bit versions of JRE
and FFmpeg. 64-bit forks of the AviSynth project do exist, however.

Frame accuracy: Currently, FFmpeg decoding in jVQA is not frame accurate. To
avoid frame synchronization errors, provide AviSynth scripts as input instead
of video files.

Content support: jVQA is designed for video content in Y'V12 colorspace, as the
standard for MPEG-based video. Only the Y plane is processed, which is expected
to provide 8-bit pixels. Currently, no resolution limitations for jVQA are
known besides memory requirements: 720p HD content requires 384 MB to be
allocated to the JVM (provide the parameter "-Xmx384m"); 1080p Full HD content
requires 640 MB allocated; and 4K content requires 1024 MB allocated.
Operations based on an integral image, such as down-sampling and Box filtering,
are currently limited to 8-bit 4K UHD content due to the basic 32-bit integer
type employed.


LEGAL STATEMENTS
================

Copyright 2014-2015 Estêvão Chaves Monteiro

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

[Hyral]

The paper presented at IEEE VCIP 2015, "Perceptual video quality assessment in adaptive streaming encoding", is now available at my home page at http://estevao.altervista.org.

[benwaggoner]

Does this provide intra-frame only statistics, or does it support any inter-frame analysis? For example, stSSIM?

It's definitely a worthy project!

[Hyral]

I appreciate very much your consideration, Mr. Waggoner, thank you!

For now, only intra-frame statistics are implemented. I am aware of stSSIM, as well as Speed SSIM, MOVIE, and VQM (Wolf & Pinson, 2004), to name a few relevant inter-frame metrics. In this first stage, I focused on efficiency, i.e., those metrics that achieve high DMOS correlation with relatively low computational complexity and low memory usage.

I am very much interested in expanding to more statistics. At the moment, though, I am focused on finishing my M.Sc. thesis and publishing papers to expand upon the VCIP one, while also having my day job, but I hope to get back to jVQA soon.

[Hyral]

Beta version 10 is released, with automatic AviSynth script generation, better file selection, external defaults configuration with properties file (mostly for default file directory selection) and higher quality default metric (higher DMOS correlation).