Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 30th March 2009, 15:21   #21  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
G_M_C: How do you tell if one video looks better than another, if you had nothing but those two videos on hand? You'd watch them, and say which one looks better to you. Same idea applies here, only with more restrictions to prevent skewing from having extra knowledge (which video was encoded with what settings, etc).
__________________
You can't call your encoding speed slow until you start measuring in seconds per frame.
Sagekilla is offline   Reply With Quote
Old 30th March 2009, 15:33   #22  |  Link
G_M_C
Registered User
 
Join Date: Feb 2006
Posts: 1,076
Quote:
Originally Posted by Sagekilla View Post
G_M_C: How do you tell if one video looks better than another, if you had nothing but those two videos on hand? You'd watch them, and say which one looks better to you. Same idea applies here, only with more restrictions to prevent skewing from having extra knowledge (which video was encoded with what settings, etc).
A "double blind" (*)test/screening you mean ? But you still need a fair number of people to make the test statistically sigificant.

(*)[funny]However the "Blind" part of it doesnt seem appropriate [/funny]
G_M_C is offline   Reply With Quote
Old 30th March 2009, 18:36   #23  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
No, doesn't necessarily have to be double blind. Yes, it is helpful to have the extra degree but you can get away with the testers knowing which video is which.
__________________
You can't call your encoding speed slow until you start measuring in seconds per frame.
Sagekilla is offline   Reply With Quote
Old 30th March 2009, 19:19   #24  |  Link
Sharktooth
Mr. Sandman
 
Sharktooth's Avatar
 
Join Date: Sep 2003
Location: Haddonfield, IL
Posts: 11,768
blind test is the way to go for testing quality, since as i've said, metrics are juts... metrics. that means they measure differencies between samples weigthing them in some way. but the fact is, a better metric doesnt represent better visual quality.
lets make an example:
sample A: high quality picture
sample B: mid quality picture
sample C: low quality picture

all pictures represent the same image... just with different quality.
picture B is our source for comparation.

results for metrics>
picture A: low metric.
picture B: highest metric.
picture C: low metric.
conclusion: picture B has the highest metric. picture A even if it is the highest quality picture of the pack, is rated "low" coz it differs from picture B that is our source for comparation and has a low metric...

results for human eye>
picture A: highest quality
picture B: mid quality
picture C: worst quality
conclusion: if you look at the pictures with your eyes you will have no doubt that A is the best one... while metrics are telling you something else...

so, all in all, this is the demonstration that metrics do not represent quality and you must never trust them if you compare different encoders.
that's also the reason why elecard has higher psnr than x264 but x264 produces a much higher visual quality (and some "smart" people produce docs with graphs but without any kind of visual comparison...).

Last edited by Sharktooth; 30th March 2009 at 19:32.
Sharktooth is offline   Reply With Quote
Old 30th March 2009, 19:51   #25  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,489
Blind test don't work for video simply it's really hard to evaluate overall quality for long sequence (simple example is VBR vs CBR for the same codec). In practice blind test for video are generally less accurate than metric. It's like that. Speak about that with developper ... they don't trust generaly blind test for video. I have never see even here on doom9 really reliable blind test ...
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 30th March 2009, 23:47   #26  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,489
Quote:
that's also the reason why elecard has higher psnr than x264 but x264 produces a much higher visual quality (and some "smart" people produce docs with graphs but without any kind of visual comparison...).
It's simply false ... for all affirmation ... lol

1) x264 without psy (AQ and SSD) produce better metric than Mainconcet SDK without psy (AQ and FGO) and with relative large margin. x264 is in fact the best in area for metric test ... ;-)
x264 use simply psy tools by default and not Mainconcept.

2) Some people even here on doom9 forum find that Mainconcept SDK produce better visual result than x264. The principal particulary for HVS is it's ... subjective ... and by definition you can't contradict that. In fact IMO x264 and Mainconcept SDK produce in most case comparable visual result and IMO you can notice real difference only for really particular sequences at really low quality encoding.

3) graph and metric are usefull for particular test like speed test simply because you must have really reliable quality reference. Make speed test with subjective comparison and without metric test is simply impossible. No way. It's like that. No possible discution here. Final point.
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9

Last edited by Sagittaire; 30th March 2009 at 23:58.
Sagittaire is offline   Reply With Quote
Old 31st March 2009, 02:50   #27  |  Link
Sharktooth
Mr. Sandman
 
Sharktooth's Avatar
 
Join Date: Sep 2003
Location: Haddonfield, IL
Posts: 11,768
Quote:
Originally Posted by Sagittaire View Post
Blind test don't work for video simply it's really hard to evaluate overall quality for long sequence (simple example is VBR vs CBR for the same codec). In practice blind test for video are generally less accurate than metric. It's like that. Speak about that with developper ... they don't trust generaly blind test for video. I have never see even here on doom9 really reliable blind test ...
in practice blind tests made lame the best mp3 encoder out there.
that is the proof blind tests work for encoders development.

Quote:
Originally Posted by Sagittaire View Post
It's simply false ... for all affirmation ... lol

1) x264 without psy (AQ and SSD) produce better metric than Mainconcet SDK without psy (AQ and FGO) and with relative large margin. x264 is in fact the best in area for metric test ... ;-)
x264 use simply psy tools by default and not Mainconcept.

2) Some people even here on doom9 forum find that Mainconcept SDK produce better visual result than x264. The principal particulary for HVS is it's ... subjective ... and by definition you can't contradict that. In fact IMO x264 and Mainconcept SDK produce in most case comparable visual result and IMO you can notice real difference only for really particular sequences at really low quality encoding.

3) graph and metric are usefull for particular test like speed test simply because you must have really reliable quality reference. Make speed test with subjective comparison and without metric test is simply impossible. No way. It's like that. No possible discution here. Final point.
1) there are no doubts x264 - if we talk about quality - is better than mainconcept h.264 encoder

2) psy opts are hardly subjective. if you have a source with grain, you expect the encoder to keep the grain... otherwise you filter it out before encoding. another point is artifacts. you dont want them if they're not on the source

3) i disagree since, as i said, metrics do not represent quality in any way.

Last edited by Sharktooth; 31st March 2009 at 03:33.
Sharktooth is offline   Reply With Quote
Old 31st March 2009, 03:10   #28  |  Link
Shinigami-Sama
Solaris: burnt by the Sun
 
Shinigami-Sama's Avatar
 
Join Date: Oct 2004
Location: /etc/default/moo
Posts: 1,923
metrics are good for sanity tests
for quality I'll stick to my eyes
__________________
Quote:
Originally Posted by benjust View Post
interlacing and telecining should have been but a memory long ago.. unfortunately still just another bizarre weapon in the industries war on image quality.
Shinigami-Sama is offline   Reply With Quote
Old 31st March 2009, 04:11   #29  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
Metrics are useful for comparing how similar your image is to the source image. That's about it. Unfortunately it's metric optimal to do a number of visually poor things like prefer a soft image.

With that said, they can be useful for providing a rough speed vs quality tradeoff in an encoder, or among encoders, as long as psy optimizations are disabled. But, any serious quality comparison should be done with psy-rd and subjectively, not with metrics.
__________________
You can't call your encoding speed slow until you start measuring in seconds per frame.
Sagekilla is offline   Reply With Quote
Old 31st March 2009, 08:32   #30  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,489
Quote:
in practice blind tests made lame the best mp3 encoder out there.
that is the proof blind tests work for encoders development.
Video don't work like Audio. Audio algo are massively based on psychoacoustic model and not video codec. Make psnr test with audio (direct comparison between source source and encoding) is non sens here but not for video. Anyway mp3 codec is simply mathematical algorithm and you can always evaluate efficacity with metric if you use metric based on psychoaccoustic model. Try to find good blind test for video and speak after about that.


Quote:
1) there are no doubts x264 - if we talk about quality - is better than mainconcept h.264 encoder
No doubt only for your eyes. No doubt for psnr (delta is generaly at 0.3-0.5 dB in my test). IMO with good setting (aka psy tools) mainconcept provide generaly similar quality to x264.


Quote:
2) psy opts are hardly subjective. if you have a source with grain, you expect the encoder to keep the grain... otherwise you filter it out before encoding. another point is artifacts. you dont want them if they're not on the source
Not always. x264 psy tools produce catastrophic result in some case (like for anime for example). For many people grain is artefact by itself and for these people codec with low grain preservetion produce good visual result and better visual experience than the source ...


Quote:
3) i disagree since, as i said, metrics do not represent quality in any way.
It's false. You don't know simply how work metric and how psy tools change metric. If you don't trust metric you can't simply make speed comparison. It's like that. Final point.
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 31st March 2009, 10:27   #31  |  Link
Gabriel_Bouvigne
L.A.M.E. developer
 
Gabriel_Bouvigne's Avatar
 
Join Date: Dec 2001
Location: Paris - France
Posts: 276
Quote:
Originally Posted by Sagekilla View Post
No, doesn't necessarily have to be double blind. Yes, it is helpful to have the extra degree but you can get away with the testers knowing which video is which.
Subjective testing MUST be double blind. You can not trust results of people who know which codec is used for a specific candidate.

Quote:
Originally Posted by Sagittaire View Post
Blind test don't work for video simply it's really hard to evaluate overall quality for long sequence (simple example is VBR vs CBR for the same codec). In practice blind test for video are generally less accurate than metric. It's like that. Speak about that with developper ... they don't trust generaly blind test for video. I have never see even here on doom9 really reliable blind test ...
DBT work for video. There is no more issue related to long content within video than within audio tests, and your example of CBR vs VBR for the same codec is not more an issue for video codecs than it is for audio codecs.
The main point is simply that objective grading is way cheaper and faster than subjective grading, thus as long as you can manage to reliably use objective grading it is a better choice to use it.

Quote:
Originally Posted by Sagittaire View Post
Video don't work like Audio. Audio algo are massively based on psychoacoustic model and not video codec. Make psnr test with audio (direct comparison between source source and encoding) is non sens here but not for video. Anyway mp3 codec is simply mathematical algorithm and you can always evaluate efficacity with metric if you use metric based on psychoaccoustic model. Try to find good blind test for video and speak after about that.
This is because until now, there were enough progress made on video encoder just by using brute force. Pure brute force (like extensive RD) and mathematical improvements are less risky that any psychoacoustic or psychovisual trick.
In no way this means that audio coding is more inherently dependent on psy models than video coding is.

Now, about the point that mp3 would be a simple mathematical algorithm, that is not more the case for mp3 than for h.264.
Objective grading of any psy-model based encoder (audio or video) requires the use of another psy-model. The big caveat is that the model used for evaluation must then be fully reliable. If it was fully reliable, that would mean that a fully reliable model would exist, and then you could simply put that model within your encoder and that would result into a fully reliable psy-model based encoder.
I am sorry, but until now such a beast doesn't exist.
State of the art objective grading of audio based on a psymodel only achieve about 60% of correlation with human results. While a 60% correlation is a significant achievement, it is still far from being able to totally replace DBT.

Something like a simple naive PSNR computation, which is throw away by a simple +1 to every sample or any ROI based optimisation, can not be considered as an overall good quality estimator.
It can only be used in some specific, controlled, and limited cases.
Gabriel_Bouvigne is offline   Reply With Quote
Old 31st March 2009, 10:46   #32  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,688
Quote:
Originally Posted by Gabriel_Bouvigne View Post
Subjective testing MUST be double blind. You can not trust results of people who know which codec is used for a specific candidate.
Even that may not be enough. An experienced and well-informed viewer can almost certainly tell an x264 from a Mainconcept stream, for example, especially at low bitrates.

You'd basically have to restrict yourself to viewers and testers who are unable to identify which stream comes from which encoder.
Dark Shikari is offline   Reply With Quote
Old 31st March 2009, 10:55   #33  |  Link
Gabriel_Bouvigne
L.A.M.E. developer
 
Gabriel_Bouvigne's Avatar
 
Join Date: Dec 2001
Location: Paris - France
Posts: 276
Quote:
Originally Posted by Dark Shikari View Post
Even that may not be enough. An experienced and well-informed viewer can almost certainly tell an x264 from a Mainconcept stream, for example, especially at low bitrates.

You'd basically have to restrict yourself to viewers and testers who are unable to identify which stream comes from which encoder.
That's true, as if you keep those subjects your test is not double blind anymore (test subject being able to know which encoder is used)

Btw, for those who think that DBT are useless for video coding, ITU doesn't seem to agree on that:
ITU-R BT.500-11
ITU-R BT.700
Gabriel_Bouvigne is offline   Reply With Quote
Old 31st March 2009, 12:17   #34  |  Link
Chengbin
Registered User
 
Join Date: Oct 2007
Posts: 1,060
Quote:
Originally Posted by Dark Shikari View Post
Even that may not be enough. An experienced and well-informed viewer can almost certainly tell an x264 from a Mainconcept stream, for example, especially at low bitrates.

You'd basically have to restrict yourself to viewers and testers who are unable to identify which stream comes from which encoder.
Wow.

I'm curious about your "especially at low bitrates" comment. What characteristics are there? Is one better than the other or something?
Chengbin is offline   Reply With Quote
Old 31st March 2009, 13:43   #35  |  Link
Esurnir
Registered User
 
Join Date: Jun 2008
Posts: 143
Quote:
Originally Posted by Chengbin View Post
Wow.

I'm curious about your "especially at low bitrates" comment. What characteristics are there? Is one better than the other or something?
--Touhou-RD 1:0

The difference is striking.

Last edited by Esurnir; 31st March 2009 at 13:47.
Esurnir is offline   Reply With Quote
Old 31st March 2009, 13:49   #36  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,489
Quote:
Originally Posted by Esurnir View Post
No ... it's more subtil ... and this comparison doen't mean anything because it's possible to obtain really better result for Mainconcept SDK, Nero or VP7.
The only good test is for touhou (x264 is able to reproduce very well static part with high complexity) but it's a really and too specific sample.
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9

Last edited by Sagittaire; 31st March 2009 at 13:56.
Sagittaire is offline   Reply With Quote
Old 31st March 2009, 17:13   #37  |  Link
Shinigami-Sama
Solaris: burnt by the Sun
 
Shinigami-Sama's Avatar
 
Join Date: Oct 2004
Location: /etc/default/moo
Posts: 1,923
if that's not obvious across the board its time to invest in some glasses, and a new monitor
__________________
Quote:
Originally Posted by benjust View Post
interlacing and telecining should have been but a memory long ago.. unfortunately still just another bizarre weapon in the industries war on image quality.
Shinigami-Sama is offline   Reply With Quote
Old 31st March 2009, 17:15   #38  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,688
Quote:
Originally Posted by Chengbin View Post
Wow.

I'm curious about your "especially at low bitrates" comment. What characteristics are there? Is one better than the other or something?
AQ and especially psy-RD have a very distinct visual signature, especially when one can compare to an encode without them.
Dark Shikari is offline   Reply With Quote
Old 31st March 2009, 17:17   #39  |  Link
Sharktooth
Mr. Sandman
 
Sharktooth's Avatar
 
Join Date: Sep 2003
Location: Haddonfield, IL
Posts: 11,768
well, gabriel already said everything that was needed to say.
however:
Quote:
o doubt only for your eyes. No doubt for psnr (delta is generaly at 0.3-0.5 dB in my test). IMO with good setting (aka psy tools) mainconcept provide generaly similar quality to x264.
this is meaningless since i stated "if we speak about quality", and a metric measure is not about quality...
and your "final point" is pointless since there is no perfect mathematical model of the human eye and if you think about psy opts, they exists since metrics cant replace visual perception...
Sharktooth is offline   Reply With Quote
Old 31st March 2009, 19:22   #40  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,489
Quote:
Originally Posted by Sharktooth View Post
this is meaningless since i stated "if we speak about quality", and a metric measure is not about quality...
and your "final point" is pointless since there is no perfect mathematical model of the human eye and if you think about psy opts, they exists since metrics cant replace visual perception...
psy tool for x264 are metric tools by itself ... lol

- AQ measure complexity block
- psy rdo use simply SSD metric for choose better RD decision.

I have impression when I read your thread that psy for x264 is like magical tools. Psy tools use simply other metric tools for better complexity/texture detection.
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9

Last edited by Sagittaire; 31st March 2009 at 19:27.
Sagittaire is offline   Reply With Quote
Reply

Tags
codecs, comparison, h.264/avc

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 21:52.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.