Zopti

[Iron_Mike]

Originally Posted by ChaosKing

https://github.com/theChaosCoder/zoptilib
Mdsi and butteraugli no longer contaminate the clip. That means metrics=['ssim', 'gmsd', 'mdsi', 'butteraugli'] is the same as metrics=['mdsi', 'butteraugli', 'ssim', 'gmsd'] now.
This also means that you won't see the heatmap...

could you elaborate on that ?

is this in regards to the RGB conversion for MDSI/Butteraugly ?

Also, running Butteraugly via Zopti takes quite a while on my system... way longer than VMAF, GMSD, SSIm etc... is this normal ?

Any settings to speed this up ?

Thanks.

[ChaosKing]

Yes, it doesn't pass the rgb converted clip but instead it copies the clip props to the old clip now.
Butteraugli is sloooow, can't do much about it.

[Iron_Mike]

Originally Posted by ChaosKing

Yes, it doesn't pass the rgb converted clip but instead it copies the clip props to the old clip now.
Butteraugli is sloooow, can't do much about it.

okay, so if I run metrics=['ssim', 'gmsd', 'mdsi', 'butteraugli'] in the same Zopti call, it will convert the clips to RGB for MDSI/Butter but will do GMSD/SSIM calc in YUV ?

[ChaosKing]

Originally Posted by Iron_Mike

okay, so if I run metrics=['ssim', 'gmsd', 'mdsi', 'butteraugli'] in the same Zopti call, it will convert the clips to RGB for MDSI/Butter but will do GMSD/SSIM calc in YUV ?

Yes and yes and the "auto-rgb-convert" is still present for mdsi and butter.

[Iron_Mike]

does SSIM/GMSD via Zopti/muvsfunc upscale the chroma planes to the src resolution of the clip if we specifiy planes=1|2 (e..g is src is 444 but dist is 420) ?

[WorBry]

No it does not. That was the whole point of my last post in the 'SSIM/GMSD metrics thread':

https://forum.doom9.org/showthread.p...63#post1870863


You posted the same question there and before I could answer, deleted it and are now asking it here. What gives ?

No resize or weights are applied to the 'processed-as-luma' chroma planes.

Originally Posted by Iron_Mike

e..g is src is 444 but dist is 420) ?

That combination would throw an error irrespective of the plane being processed.

 "clip1" and "clip2" must be of the same format!

[Iron_Mike]

Originally Posted by WorBry

No it does not. That was the whole point of my last post in the 'SSIM/GMSD metrics thread':

https://forum.doom9.org/showthread.p...63#post1870863

You posted the same question there and before I could answer, deleted it and are now asking it here. What gives ?

what gives ?

well, Brian... take a really hard guess why I opted to move this to the ZOPTI thread instead of leaving it in the GMSD thread.... ? considering we're using Zopti as a wrapper...


in any case - back to the point - so in case of GMSD/SSIM scoring planes 1/2 of a 444 src vs 420 dist, is Zopti/muvsfunc downscaling the src to the dist resolution or does the GMSD/SSIM metric not require both clips to be in the same resolution ?

[WorBry]

I repeat...

Originally Posted by WorBry

That combination would throw an error irrespective of the plane being processed.

 "clip1" and "clip2" must be of the same format!

[Iron_Mike]

Originally Posted by WorBry

I repeat...

yeah, that's why I ask here in the Zopti thread...

I just ran a 444 ref against a 420 dist, scored GMSD/SSIM in planes 1|2 and it did not throw an error (meaning: chroma resolution of src/dist was NOT the same)... also (as far as I understand it), vs.format does not include src/Y resolution...

so, question remains...

[ChaosKing]

Originally Posted by Iron_Mike

yeah, that's why I ask here in the Zopti thread...

I just ran a 444 ref against a 420 dist, scored GMSD/SSIM in planes 1|2 and it did not throw an error (meaning: chroma resolution of src/dist was NOT the same)... also (as far as I understand it), vs.format does not include src/Y resolution...

so, question remains...

Could you provide your script? I always get "clip1" and "clip2" must be of the same format!

My test

#c is 420
a = mvf.GetPlane(c, 1)
c = core.fmtc.resample(c, css="444") 
b = mvf.GetPlane(c, 1)#.fmtc.resample(css="444") 

c = muf.SSIM(a,b)

[WorBry]

Originally Posted by ChaosKing

Could you provide your script? I always get "clip1" and "clip2" must be of the same format!

Quite !

[Iron_Mike]

Originally Posted by ChaosKing

Could you provide your script? I always get "clip1" and "clip2" must be of the same format!

my bad, u're right - I have a block of code there for SSIM/GMSD (via Zopti) that auto-converts dist to src format in case format differs - to avoid that exact error... ;-)

so what are opinions about how to weigh the GMSD/SSIM results if the css of the dist differs ?

[WorBry]

Originally Posted by Iron_Mike

so what are opinions about how to weigh the GMSD/SSIM results if the css of the dist differs ?

You don't. You either convert the css of the 'distorted' clip to that of the reference or vice versa. They must be the same format - period. It's the fundamental requirement of Full Reference Image Quality Assessment. You seem to be laboring under this notion that it should somehow be different. 'Full Reference' - think about it.

[Iron_Mike]

Originally Posted by WorBry

You don't. You either convert the css of the 'distorted' clip to that of the reference or vice versa. They must have the same format - period. It's the fundamental requirement of Full Reference Image Quality Assessment. You seem to be laboring under this notion that it should somehow be different.

well Brian, a "fundamental requirement" is to read posts properly... or to put 1+1 together

I was referring to weighing the chroma plane results (if dist had different css than ref) when combining them with the Y plane result for a total GMSD/SSIM score based on all three planes...............

in addition... chroma plane results can also be (optionally) weighed (in a combined score) even if both ref/dist have the exact same css considering the fact that color drift has less perceived impact in human vision...


whatever it is that u're mumbling in regards to "format" has nothing to do with any of it... just because u're matching a format so u can run the metric does not mean the data is (or should be considered) equal........

[WorBry]

Someone else deal with this please. I'm done with this nonsense.

Also Iron_Mike, please desist in referring to me by anything other than my forum user name. I don't know what you are trying to convey, but it's rather creepy and smacks of trollism.

[ChaosKing]

Originally Posted by Iron_Mike

well Brian, a "fundamental requirement" is to read posts properly... or to put 1+1 together

I was referring to weighing the chroma plane results (if dist had different css than ref) when combining them with the Y plane result for a total GMSD/SSIM score based on all three planes...............

But to compare different css you have to resize the chroma plane to match the same clip format for ssim/gmsd. It's similar to a rgb and yuv clip, one has to be converted wich alters your final score a little bit.

hmm a score based on all three planes... For that I tried to see how the video looks like if you extremely blur the chroma plane. Surprisingly only the colors are a bit off.
I guess the final score could be as simple as Plane1 80% + P2 10% + P3 10% ... I don't know what a good weighted total score should look like.

c2_1 = mvf.GetPlane(c2, 1).std.BoxBlur(hradius=5, vradius=5)
c2_2 = mvf.GetPlane(c2, 2).std.BoxBlur(hradius=5, vradius=5)
c2 = core.std.ShufflePlanes(clips=[mvf.GetPlane(c2, 0), c2_1, c2_2], planes=[0, 0, 0], colorfamily=vs.YUV)

a = mvf.GetPlane(c, 1)
b = mvf.GetPlane(c2, 1)
#a=c 
#b=c2
c = muf.SSIM(b,a).text.FrameProps()
c.set_output()

p.s the ssim score for the blurred chroma plane is 0.8

[WorBry]

Originally Posted by ChaosKing

I don't know what a good weighted total score should look like.

I'm not sure. As mentioned in the other 'SSIM: GMSD metric thread', Zorr looked at the code for the AVISynth and FFMPEG SSIM metrics to see how they derive an overall score for the Y, U and V channels:

Originally Posted by WorBry

Zorr checked out the code for ffmpeg SSIM (AVISynth SSIM also) earlier in the thread. In essence, it's akin to the 'fast' SSIM metric in the MSU Quality Tool:

https://forum.doom9.org/showthread.p...30#post1866030

https://forum.doom9.org/showthread.p...62#post1866162

Originally Posted by zorr

The Avisynth version is not doing the gaussian kernel at all - it's implemented using summed area tables. That's a faster but lower quality way to calculate it, the MSU Quality measurement tool page has an example of the difference.

.....Avisynth SSIM has a plane argument which defaults to 0 and then it returns a weighted sum of the luma and chroma channels:
(0.8 * Y) + (0.1*(U+V))

But the AVISynth version is YV12 only.

Originally Posted by zorr

I looked at the ffmpeg source. It's doing the fast version too - no gaussian kernels there. The total SSIM takes into account all the planes but the weighting is different, each plane is scaled by the resolution is has. So for example with YUV420 the color planes have 4 times smaller weight. The constants 0.01 and 0.03 appear in the code and I assume that means the weights k1 and k2 are the same as in other implementations.

And:

To improve speed, this implementation uses the standard approximation of overlapped 8x8 block sums, rather than the original gaussian weights

https://github.com/FFmpeg/FFmpeg/blo...lter/vf_ssim.c

So the ffmpeg variant applies different weights according to pixel format of the clips being tested, but they must have the same pixel format and resolution

https://ffmpeg.org/ffmpeg-filters.html#ssim

Of course the FFMPEG metric can also be run on RGB clips, generating scores for the individual R,G,B channels and an overall aggregate score. But it's still using the 'fast' and less precise 'standard approximation of overlapped 8x8 block sums'.

Is there provision in the original Matlab SSIM code for measuring and weighting the chroma or some other reference where this is defined - I don't know ?

I checked out the demo 'Pro' version of the MSU-QMT, btw - the demo version intentionally generates incorrect results as a kind of 'metric watermark' but it shows the options that are available for the different measures. Looks like the 'precise' (gauss blur) and 'fast' SSIM variants are measuring Y luma only.

And then there are these other 'SSIM algorithm-based' variants - DSSIM and SSIMULACRA - intended (like Butteraugli) primarily for measuring differences in still images, which process in CIE Lab* color space and apply weights to the L, a and b channels to derive an overall score:

https://github.com/kornelski/dssim

https://github.com/cloudinary/ssimulacra

https://cloudinary.com/blog/detectin...ing_ssimulacra

Personally, I'd be more interested in deriving a composite SSIM score for the combined chroma channels, rather than (or in addition to) an 'overall' score, but that would also require consideration of the weightings.

[VS_Fan]

Originally Posted by WorBry

... Personally, I'd be more interested in deriving a composite SSIM score for the combined chroma channels, rather than (or in addition to) an 'overall' score

How about stacking (horizontally or vertically) the u and v planes in vapoursynth, then generate YUV/Y4M files so you can obtain SSIM metrics by using any of the tools mentioned?

I would expect the result to be very close to averaging the results of the individual u and v planes.

[WorBry]

Now that's a novel idea. I was thinking of maybe merging the luma-converted U and V planes.

[VS_Fan]

With the same general idea, one could obtain an “an 'overall' score” by stacking all the planes:

ret = some_video_source
#This should work to preserve vertical resolution, and stacking all Y,U and V, no matter the css (chroma subsampling: 411, 420, 422, 444)
retc = core.std.StackHorizontal(clips=[retU,retV]) if ret.format.subsampling_h == 0 else core.std.StackVertical(clips=[retU,retV])
ret = core.std.StackHorizontal(clips=[retY,retc])

I would also expect this ‘overall’ result to be close to that of applying the weights as you exposed they do in ffmpeg: “each plane is scaled by the resolution it has”, and I guess they are something like:

sub-		Y 		U 		V
sampling	ratio		ratio		ratio

4:1:1		2/3		1/6		1/6 
4:2:0		2/3		1/6		1/6 
4:2:2		1/2		1/4		1/4 
4:4:4		1/3		1/3		1/3