Is x265 fundamentally broken?

[Grojm]

Originally Posted by benwaggoner

Reading through the readthedocs, it's obvious how quality is different between presets, because different modes controlled by different presets use different ways to measure and optimize quality.

Presets control --rd, for a big example. What even gets included in rate distortion optimization changes with rd level. So how chroma quality is measured and optimized changes between 2 and 4. Something with complex chroma running at a fast preset might give smaller size because it is just making the chroma look terrible.

aq-mode also has fundamental impact on how the codec works in practice.

THIS was the missing piece of information I needed. Thanks!


Because I always thought, if you give the encoder more time (slower preset) to try different encoding settings, at least the efficiency will stay the same or improve. So, if you keep the quality metric fixed (howsoever you define your metric, doesn't matter), the resulting file size will be at least the same or smaller.
But, if with a slower preset, also the metric itself changes, of course this is not the case any more.

I haven't thought about how the preset changes how the metric is calculated itself.

However, I think there should be found a way to keep the metric fixed for different presets. This would avoid a lot of confusion.

[benwaggoner]

Originally Posted by Grojm

However, I think there should be found a way to keep the metric fixed for different presets. This would avoid a lot of confusion.

The problem is that useful metrics are more expensive to calculate, and so aren't going to be used in faster presets. And there isn't any way to set a metric directly when encoding in a useful mode. Sure you could make an encoder that would encode to exactly 30 PSNR, but it would do it by making every frame and likely every pixel that, which would be a horribly suboptimal way to encode visually.

What we want an encoder to do is to provide optimal subjective quality, and that can only really be tested by humans in a double-blind environment. VMAF attempts to estimate what a human would rate content, and it's the least-bad metric available to the public. But it falls short in many ways, doesn't do HDR at all, etcetera.

If we had a perfect objective metric, all encoders would do is optimize for that. Alas, things are vastly more complicated.