VP8 vs x264

[CruNcher]

Emanem_
if you would really compare it on a psy level VP8 currently has only 1 real Psy like option and that's a very basic deadzone change done via --sharpness=x (which could have been usefull in this case)
also posting non correct frame comparisons with frames that are different between source and encodes doesn't help much (also not informing what frames we actually see their and how far they away from the keyframe isn't nice)

And yeah it's obvious you didn't hit bitrate with ffmpeg and posting this then as valid comparison is blatantly wrong you should know that

read this http://pastebin.org/346660 understand it live it

[Dark Shikari]

Originally Posted by CruNcher

Emanem_
if you would really compare it on a psy level VP8 currently has only 1 real Psy like option and that's a very basic deadzone change done via --sharpness=x (which could have been usefull in this case)
also posting non correct frame comparisons with frames that are different between source and encodes doesn't help much (also not informing what frames we actually see their and how far they away from the keyframe isn't nice)

And yeah it's obvious you didn't hit bitrate with ffmpeg and posting this then as valid comparison is blatantly wrong you should know that

read this http://pastebin.org/346660 understand it live it

Come on, don't post my drafts, post the link to the actual blog post

[CruNcher]

Yeah sorry credits go to Dark and all the other contributors in #x264

[Emanem_]

Originally Posted by Dark Shikari

Come on, don't post my drafts, post the link to the actual blog post

Thanks for your ideas and reccomendations to properly test videos.
How would you objectively test codecs with psy on? Which metric would you reccommend?

OMG, I posted that table with 'real' bitrate and thought that encoders would have used the requested bitrate!

As soon as I have time I'll reupdate and republish it.

Regarding objective measures, do you have any metric to suggest other than PSNR and SSIM?
Cheers,

[Dark Shikari]

Originally Posted by Emanem_

Thanks for your ideas and reccomendations to properly test videos.
How would you objectively test codecs with psy on? Which metric would you reccommend?

That's inherently not possible -- I'd use screenshots and my eyes.

Originally Posted by Emanem_

Regarding objective measures, do you have any metric to suggest other than PSNR and SSIM?
Cheers,

Not that I know of. There's some fancy ones, but I don't know of any implementations. Or any proof that they're any good.

Here's what I'd do:

1. Pick a metric (SSIM or PSNR).

2. Tune all tested encoders, wherever possible, for these metrics (e.g. --tune psnr in x264). Use the slowest settings and so forth -- get the most out of each encoder.

3. Do an objective chart accordingly.

4. Tune the encoders for psy instead, and use that to generate screenshots.

The downsides of this method are that:

a) The graphs will bias against encoders that have good psy optimizations, like x264. This is because they don't take into account the fact that psy optimizations make the difference between encoders greater than one would measure via raw objective metrics.

b) The graphs will bias even more against encoders that can't turn off their psy optimizations (e.g. if you're optimizing for PSNR, Theora Ptalarbvorm is by-default-and-you-can't-turn-it-off optimized for SSIM). This is to the point where if you can't disable psy opts, objective testing is basically unfair and shouldn't be done.

[Emanem_]

Originally Posted by Dark Shikari

That's inherently not possible -- I'd use screenshots and my eyes.Not that I know of. There's some fancy ones, but I don't know of any implementations. Or any proof that they're any good.

Thanks again!

I do understand your points.
But how must be hard to develop a codec and test/verify that what you've done is an improvement if there is no such automated way to do it? Do you authors of x264 have at your disposition a group of experts to help you with psy measurements?

Anyway, about x264 and VP8, I've always used the slowest settings (think during encoding that VP8 in the Shark scene goes 1.5 FPS while x264 30+ on my Phenom II X4 965 B.E.).

Apart the stupid mistake (not taking a look at the table with file size and real bitrates), I've always tried to compare apples to apples (i.e. 4 MBps bitrate) and I tried to do a visual comparison as well.
And, with surprise, now that I look at that table again, VP8 bitrate for 4 MBps is lower than x264! So could it have used more quality? Perhaps those 40 KBps more in the first Avatar scene could decrease distance with x264 HP?

Indeed my comparison didn't pretend to be serious (it's in the title).

But at the end of the day I've found out that VP8 is very good quality wise.
Have you ever give it a shot, you know just out of curiosity, to take a peek at the competition? Have you ever encoded a HD video with VP8 and seen it? I'm just curious...

I mean, clearly in the eyes of an expert my comparison lacks of maybe ppm (or png) frames (btw should I be authorized to publish them if I don't own the images?) and the final encodings, plus maintaining a strict filesize between different encodings.
But you see, I've always tried to be more fair than possible.

At the end of the day I look to the better codec as the codec which given the same bitrate will provide more quality or if same quality least disadvantages (i.e. no software patents and/or speed).

Thanks again for your hints,
Regards,

Ps. I'm not against x264. I don't like the fact that H.264 is patented (and generally software patents do exist). Nevertheless I still think x264 is the best and stuff encoded with it works on my PS3 beautifully
I have never taken a look at x264 sources, but I bet the quality is as the software itself, really and truly exceptional. Too bad for patents.

[Dark Shikari]

Originally Posted by Emanem_

Thanks again!

I do understand your points.
But how must be hard to develop a codec and test/verify that what you've done is an improvement if there is no such automated way to do it? Do you authors of x264 have at your disposition a group of experts to help you with psy measurements?

We have our eyes and intuitions But of course I don't claim to have "golden eyes" or any of that type of BS that audiophiles like to spout; I'm as limited as anyone else.

Originally Posted by Emanem_

Anyway, about x264 and VP8, I've always used the slowest settings (think during encoding that VP8 in the Shark scene goes 1.5 FPS while x264 30+ on my Phenom II X4 965 B.E.).

30+ FPS isn't the slowest . Try placebo.

Originally Posted by Emanem_

But at the end of the day I've found out that VP8 is very good quality wise.
Have you ever give it a shot, you know just out of curiosity, to take a peek at the competition? Have you ever encoded a HD video with VP8 and seen it? I'm just curious...

Yes, I tested it as part of my analysis of VP8. It came out rather lacking, losing by about 20-30% PSNR-wise and even more visually (due to lack of psy opts). It beat x264 Baseline profile by about 8% though, PSNR-wise. Of course, in a visual comparison, even x264 Baseline beat VP8, but that's not surprising.

It's even worse if you pick a non-super-slow mode though; VP8 does incredibly badly on its "fast" modes, which is what really matters for practical use (try encoding a 1080p movie with --best!).

Of course, speed issues are problems with the encoder, not the format. The format itself is a bit better compression-wise than H.264 Baseline, but greatly inferior to Main or High. Speed-wise, it's pretty similar to Main or High, maybe a bit slower.

[LoRd_MuldeR]

Originally Posted by Dark Shikari

Not that I know of. There's some fancy ones, but I don't know of any implementations. Or any proof that they're any good.

Well, there is the MSU Quality Measurement Tool available, which in includes an implementation of VQM along with some others.

Be aware: I have no experience how useful VQM is or how it compares to SSIM. Just mentioned it for completeness...