Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 13th December 2015, 17:13   #1  |  Link
Solon8
Registered User
 
Join Date: Mar 2013
Posts: 35
SSIM: when to use it?

Hi,

When is it correct to use the structural similarity index? Please correct the assumptions below:

It's not right to use the SSIM index to compare two different films quality. For example:
- Film A got 0.985.
- Film B got 0.970.
It doesn't necessarily mean Film A has better quality than Film B.

But is it right to use the SSIM index to compare two encoding from the same film and source? For example:
- Source A encoded with settings x got 0.9854.
- Source A encoded with settings y got 0.9716.
Is it OK to say settings x are somewhat "better", or rather, closer to the source, than settings y?

Or can you not compare two SSIM indexes at all?

If not, I do not understand the correct situations to use the SSIM index. Is it just being used by TV companies and whatnot to make sure they reach a certain index?

Please, enlighten me on SSIM!
Solon8 is offline   Reply With Quote
Old 13th December 2015, 17:18   #2  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,244
SSIM is a measure for the "error" between a reference picture and a modified version thereof. Not more, not less. SSIM gives a value of 1.0, if and only if the pictures are identical. The lower the SSIM value, the bigger the difference!

Typically the modified version is a "compressed" version of the "original" (reference) picture. So, typically, SSIM gives you a measure of how much "error" (loss) has been introduced by the compression. For video, it's simply averaged over all frames.

This also means that SSIM always needs the "original" picture, which will be used to compute the relative "error". It can not compute an absolute "quality" indicator only from the compressed/modified picture, as it is not a "blind" metric.

Finally, SSIM can be used to compare two encodes that have been created from the same source. The SSIM "error" will be computed separately for each encode (relative to the original source!) and, afterwards, the SSIM values can be compared.

Note: Whether SSIM's definition of "error" is a good approximation of the "perceived error", that is a totally different question. SSIM is believed to give a better approximation than PSNR. Still, you shouldn't rely too much on SSIM values!


Further reading:
http://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊

Last edited by LoRd_MuldeR; 13th December 2015 at 17:54.
LoRd_MuldeR is offline   Reply With Quote
Old 13th December 2015, 18:22   #3  |  Link
Solon8
Registered User
 
Join Date: Mar 2013
Posts: 35
Quote:
Originally Posted by LoRd_MuldeR View Post
SSIM is a measure for the "error" between a reference picture and a modified version thereof. Not more, not less. SSIM gives a value of 1.0, if and only if the pictures are identical. The lower the SSIM value, the bigger the difference!

Typically the modified version is a "compressed" version of the "original" (reference) picture. So, typically, SSIM gives you a measure of how much "error" (loss) has been introduced by the compression. For video, it's simply averaged over all frames.

This also means that SSIM always needs the "original" picture, which will be used to compute the relative "error". It can not compute an absolute "quality" indicator only from the compressed/modified picture, as it is not a "blind" metric.

Finally, SSIM can be used to compare two encodes that have been created from the same source. The SSIM "error" will be computed separately for each encode (relative to the original source!) and, afterwards, the SSIM values can be compared.

Note: Whether SSIM's definition of "error" is a good approximation of the "perceived error", that is a totally different question. SSIM is believed to give a better approximation than PSNR. Still, you shouldn't rely too much on SSIM values!


Further reading:
http://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf
Thank you for your reply. I'm reading that pdf right now.

Yes, SSIM's definition of error is the big question.

In the past few days, I've wanted to try several avisynth filters for noise reduction (on the same source). But I've found that some worked better in some areas and others worked better in other areas (or at least the areas I checked).

I couldn't make up my mind really, so I thought I'd check their respective SSIM index, to see if overall some performed better, or rather with less "error", than others. Bad idea?
Solon8 is offline   Reply With Quote
Old 13th December 2015, 18:29   #4  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,244
Quote:
Originally Posted by Solon8 View Post
In the past few days, I've wanted to try several avisynth filters for noise reduction (on the same source). But I've found that some worked better in some areas and others worked better in other areas (or at least the areas I checked).

I couldn't make up my mind really, so I thought I'd check their respective SSIM index, to see if overall some performed better, or rather with less "error", than others. Bad idea?
What exactly do you want to compare here?

If you compare the "denoised" version against the original source, using SSIM, then the more noise is removed the lower your SSIM index will be! That's because removing that noise creates a "difference" that will be detected by SSIM as an error (i.e. quality loss). Remember that, simply put, SSIM assumes that the original (reference) picture has "perfect quality" and that any divergence from there is an undesired "error". The conclusion that a lower SSIM means "better" denoising is also wrong!

If at all, you could compute the SSIM between original and the re-encoded clip as well as the SSIM between the original and the denoised+re-encoded clip. But I still doubt that this will give you the results you are looking for...
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊

Last edited by LoRd_MuldeR; 13th December 2015 at 18:35.
LoRd_MuldeR is offline   Reply With Quote
Old 13th December 2015, 19:35   #5  |  Link
Solon8
Registered User
 
Join Date: Mar 2013
Posts: 35
Quote:
Originally Posted by LoRd_MuldeR View Post
If you compare the "denoised" version against the original source, using SSIM, then the more noise is removed the lower your SSIM index will be! That's because removing that noise creates a "difference" that will be detected by SSIM as an error (i.e. quality loss).
That's true! I didn't think about that. Denoisers would get lower SSIM index, because it's a little "further" from the source, even if we could get better quality.

Quote:
Originally Posted by LoRd_MuldeR View Post
The conclusion that a lower SSIM means "better" denoising is also wrong!
What if instead of better, I use the term "stronger" or "more processed" denoising to qualify that?

But then I don't understand the results I had on three different sources. For each source, I did 5 sample: one with no filter and 4 others with the default noise filters in AviSynth Script Creator in MeGUI (minimal noise, little noise, medium noise and heavy noise).

Reading you I would think that the no-filter sample would get the less "errors" but it was the opposite. In each case, the no-filter one got the lower index and heavy noise the highest. Any explaination? Any interpretation?

After visual checking, the no-filter sample retained more details on close-ups (faces for example) while the noise filters smoothed the backgrounds. I would think that the grain and details in the no-filter sample would give it a better score, but that's not the case.
Maybe the smoothing of noise filters made the picture overall less blocky after re-encoding, giving them a better score?

Anyway, I'm talking about denoiser now, but I could be talking about any x264 settings. It was just a general question about SSIM comparison.

Last edited by Solon8; 13th December 2015 at 19:40.
Solon8 is offline   Reply With Quote
Old 13th December 2015, 20:04   #6  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by Solon8 View Post
But then I don't understand the results I had on three different sources. For each source, I did 5 sample: one with no filter and 4 others with the default noise filters in AviSynth Script Creator in MeGUI (minimal noise, little noise, medium noise and heavy noise).

Reading you I would think that the no-filter sample would get the less "errors" but it was the opposite. In each case, the no-filter one got the lower index and heavy noise the highest. Any explaination? Any interpretation?

After visual checking, the no-filter sample retained more details on close-ups (faces for example) while the noise filters smoothed the backgrounds. I would think that the grain and details in the no-filter sample would give it a better score, but that's not the case.
I don't think you understood what LM explained above.
What was your reference in the tests? SSIM calculation always needs a reference.
I would have thought that the "no filter" sample would be your reference but since you're talking about "errors" of that particular sample, that can't be.
What tool do you use to calculate SSIM?
Groucho2004 is offline   Reply With Quote
Old 13th December 2015, 20:10   #7  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,244
Quote:
Originally Posted by Solon8 View Post
What if instead of better, I use the term "stronger" or "more processed" denoising to qualify that?
Well, if one denoiser produces a lower SSIM value than another one (both compared to the original source), then all you know is that the one denoiser produced a "stronger" difference than the other.

Or, because SSIM tries to approximate human perception, you know that it produced a more "visible" difference.

But is that a good thing or a bad thing? Hard to say. On the one hand we want a visible difference, because we want the noise to be gone. On the other hand, we don't want any other kind of change.


Quote:
Originally Posted by Solon8 View Post
Reading you I would think that the no-filter sample would get the less "errors" but it was the opposite. In each case, the no-filter one got the lower index and heavy noise the highest. Any explaination? Any interpretation?
What did you compare? SSIM of "original source vs. encoded original source" and SSIM of "denoised source vs. encoded denoised source"?

If so, did all these encodes end up with the exactly same bitrate? Or, equivalently, did they all have the same size? If not, your comparison was biased and the SSIM values are meaningless! And even if they all did end up with the same bitrate (file size), it is clear that it is much easier for the encoder to produce something that looks similar to the denoised source (from the denoised source) than producing something that looks similar to the original source (from the original source).


Quote:
Originally Posted by Solon8 View Post
Anyway, I'm talking about denoiser now, but I could be talking about any x264 settings. It was just a general question about SSIM comparison.
With x264 the situation is more clear. You feed x264 with an original source and you get a "compressed" version as output.

So, in this case, it's clear which one is the "reference" and which one is the "modified" version. Also, it's clear that you want the "compressed" version to be as close as possible to the source - which means you want a high SSIM value.

Still, even with x264, you shouldn't rely too much on SSIM. There are "Psy" optimizations that are known to produce a worse SSIM value, but that are agreed to improve perceived quality!
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊

Last edited by LoRd_MuldeR; 13th December 2015 at 20:41.
LoRd_MuldeR is offline   Reply With Quote
Old 13th December 2015, 20:43   #8  |  Link
Solon8
Registered User
 
Join Date: Mar 2013
Posts: 35
Quote:
Originally Posted by Groucho2004 View Post
I don't think you understood what LM explained above.
What was your reference in the tests? SSIM calculation always needs a reference.
I would have thought that the "no filter" sample would be your reference but since you're talking about "errors" of that particular sample, that can't be.
What tool do you use to calculate SSIM?
Oh? Maybe I did misunderstand.

Here's what I understood: the SSIM calculation uses the source image and the reencoded image, giving a score between -1 and 1.

I'm using MeGUI. In the x264 settings, I checked "SSIM calculation" and used the --tune ssim so it disables psychovisual enhancements for the test. Said it has to.
So I do not give him a particular reference, I suppose it uses the source itself, directly. Then I just checked the logs to get the SSIM index.

I do multiple encodes to get each's SSIM index. The no-filter sample I was talking about is one of them.
I'm not saying it's the reference, but I did use it to see if noiser filters got a lower SSIM index. They should be, since they process the image more than the no-filter sample, right? But that was not the case.

Where did I get it wrong?

Quote:
Originally Posted by LoRd_MuldeR View Post
Well, if one denoiser produces a lower SSIM value than another one (both compared to the original source), then all you know is that the one denoiser produced a "stronger" difference than the other.

Or, because SSIM tries to approximate human perception, you know that it produced a more "visible" difference.

But is that a good thing or a bad thing? Hard to say. On the one hand we want a visible difference, because we want the noise to be gone. On the other hand, we don't want any other kind of change.

What did you compare? SSIM of "original source vs. encoded original source" and SSIM of "denoised source vs. encoded denoised source"?

If so, did all these encodes end up with the exactly same bitrate? Or, equivalently, did they all have the same size? If they did not have same bitrate (file size), your comparison was biased and the SSIM values are meaningless.

With x264 the situation is more clear. You feed x264 with an original source and you get a "compressed" version as output.

So, in this case, it's clear which one is the "reference" and which one is the "modified" version. Also, it's clear that you want the "compressed" version to be as close as possible to the source - which means you want a high SSIM value.

Still, even with x264, you shouldn't rely too much on SSIM. There are "Psy" optimizations that are known to produce a worse SSIM value, but that are agreed to improve perceived quality!
OK, OK!

"SSIM of "original source vs. encoded original source" and SSIM of "denoised source vs. encoded denoised source"?"

I do not understand how, in your second statement, you have "denoised source" alone. Wouldn't that mean it's been reencoded as well, if denoised?

Basically, this is what I did, tell me if it's biased :

No Filter encode:
- x264 settings: --preset veryslow --tune ssim --pass 2 --bitrate 1000 --stats ".stats" --ssim --output "output" "input"
- AVS script: nothing but loading ffms2 + FFVideoSource + Trims for the sample

"Minimal Noise" encode:
- x264 settings: --preset veryslow --tune ssim --pass 2 --bitrate 1000 --stats ".stats" --ssim --output "output" "input"
- AVS script: same as No Filter, but adding Undot()
Note: I did not choose Undot, that's the default when you choose minimal noise in MeGUI

[...]

"Heavy Noise" encode:
- x264 settings: --preset veryslow --tune ssim --pass 2 --bitrate 1000 --stats ".stats" --ssim --output "output" "input"
- AVS script: same as No Filter, but adding Convolution3D("movielq")
Note: I did not choose Convolution3D, that's the default when you choose heavy noise in MeGUI

All have the same size and the same bitrate.
Wouldn't you think, like me, that the No Filter encode, being less processed than noise filters encodes would have a higher SSIM index? It got last place.

Hence my little interpretation above, that maybe No Filter is too blocky and that, the noise filtered encodes got smoothed, thus less blocky, thus closer to the source, in some way? Maybe I should re-do the tests with a much higher bitrate and see if that's still the case?

By the way, MeGUI tells you to disable Psy optimizations during SSIM calculation. So I thought I would not use it for the test, but of course enable for the final encode with the chosen filter/settings.

Edit : I just read your edit, I'm not sure I understand the denoised source / original source, isn't the denoised source an encode too? How can you denoise a source without reencoding?
Solon8 is offline   Reply With Quote
Old 13th December 2015, 21:06   #9  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,244
If you denoise in your Avisynth script, you are denoising the source before it is sent to x264. So, x264 will encode from the the denoised source. And the SSIM value computed by x264 will be "denoised source vs. encoded denoised source".

On the other hand, if you encode the source without denoisng first, x264 will be encoding the original source. And thus the SSIM value computed by x264 will be "original source vs. encoded original source".

Consequently, the one SSIM value tells you the "error" with respect to the denoised source, while the other SSIM value tells you the "error" with respect to the original source. It's easier to achieve a low "error" with respect to the denoised version!

Therefore, it is expected that the encode made from the denoised source will have a "better" SSIM value. But does it really look better visually? That's not answered at all! And you probably won't be able to solve it with SSIM alone...
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊

Last edited by LoRd_MuldeR; 13th December 2015 at 21:12.
LoRd_MuldeR is offline   Reply With Quote
Old 13th December 2015, 21:18   #10  |  Link
Solon8
Registered User
 
Join Date: Mar 2013
Posts: 35
Quote:
Originally Posted by LoRd_MuldeR View Post
If you denoise in your Avisynth script, you are denoising the source before it is sent to x264. So, x264 will encode from the the denoised source. And the SSIM value computed by x264 will be "denoised source vs. encoded denoised source".

On the other hand, if you encode the source without denoisng first, x264 will be encoding the original source. And thus the SSIM value computed by x264 will be "original source vs. encoded original source".

Consequently, the one SSIM value tells you the "error" with respect to the denoised source, while the other SSIM value tells you the "error" with respect to the original source. It's easier to achieve a low "error" with respect to the denoised version!

Therefore, it is expected that the encode made from the denoised source will have a "better" SSIM value. But does it really look better visually? That's not answered at all! And you probably won't be able to solve it with SSIM alone...
Ohh! That makes sense. I get it now.
Well, thanks for clearing that up!

I know a bit more about SSIM now.
Solon8 is offline   Reply With Quote
Old 13th December 2015, 21:43   #11  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
There is a SSIM plugin for Avisynth which I use to compare filters, etc. Very useful.
Groucho2004 is offline   Reply With Quote
Old 14th December 2015, 11:52   #12  |  Link
Solon8
Registered User
 
Join Date: Mar 2013
Posts: 35
Quote:
Originally Posted by Groucho2004 View Post
There is a SSIM plugin for Avisynth which I use to compare filters, etc. Very useful.
Thank you, I'll check this plugin!
Solon8 is offline   Reply With Quote
Old 14th December 2015, 20:36   #13  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,244
Quote:
Originally Posted by Solon8 View Post
Thank you, I'll check this plugin!
If you want to use that for evaluating some denoise filters, I think a proper test setup would have to look like this:

Take a fairly clean source (which we consider an "optimal" reference), add some artificial noise/grain to it in order to create a "noisy" version, and then apply your denoise filters to that "noisy" version.

For each denoise filter under consideration, the SSIM would have to be calculated between the original "clean" source and the final "denoised" result.

__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊

Last edited by LoRd_MuldeR; 14th December 2015 at 22:32.
LoRd_MuldeR is offline   Reply With Quote
Reply

Tags
ssim

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:02.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2023, vBulletin Solutions Inc.