Versatile Video Coding (VVC) / H.266: HEVC successor

[ksec]

Originally Posted by Z2697

There is some diminishing return "effect" going on, like 2x more complexity and 50% less bitrate sort of thing (can't remember the exact numbers they claim). The "2x" is exponentially larger and larger and the "50%" is exponentially less and less. Not to mention that "50% less" is a bold claim especially when the fidelity is the first thing to consider.

HEVC, if I remember correctly was 8x Encode complexity with 2x Decode Complexity increase. In an Era where no one knew what TSMC is and Moore's Law was still a thing that complexity increase doesn't matter much.

VVC is complexity increase is roughly similar comparing to HEVC. So ~80x increase on encoding and 5x decoding comparing to AVC.

FVC is the outliner where it is expected to be 10x complexity in both encode and decode. We are talking about 800x increase in encoding and 50x increase in decoding comparing to AVC.

I am already questioning the feasibility of VVC ( Recent Development with M4 and Cortex X5 makes things much better ), let alone FVC.

The problem is those high bitrate % savings only comes in higher resolution and specific quality scenario. Something like You will only see the optimal 50% reduction from AVC to HEVC when you compare at 4K at sub 4Mbps. Or 1080P at below 1Mbps.

Depending on scenario, most of these results are not very practical. Even Facebook and YouTube have decided to increase the bitrate of their 1080P H.264 video to provide higher quality instead of forcing another Video Codec usage.

Much like audio, there is definitely diminishing returns going on in post-AAC LC codecs. At 256Kbps Opus may be better than AAC 256Kbps in some cases, but giving AAC ~10% headroom would have completely diminish those advantage. The question is do we still need to look at below 256Kbps usage? Even if we do, does it justify another codec if we could fix it with 20% increase in transfer and storage. Which is getting cheaper everyday. AAC-LC is also completely patent free now, and has the widest industry support apart from MP3.

The same is true with Video Codec, we need archival-grade quality at less than half the H.264 bitrate. Not Half the bitrate at below 1Mbps. i.e We have shifted to we need higher / best quality at minimal bitrate, instead of having bare minimum / good enough quality at lowest possible bitrate. The two are very different.

This is similar again in image, where JPEG XL is focusing on the higher end of quality or actual usage. According to Google Chrome 80% of picture served have bit per pixel at 1.0 or above. And JPEG XL is the best in that category. We could serve higher quality picture at same size rather than looking at bpp of 0.5.

[GeoffreyA]

You're spot on that we need archival-grade quality at a much smaller bitrate than H.264, but modern codecs aim rather for "good enough" at that smaller bitrate. Go for transparency and the bitrate scales massively, and most of the time, transparency is elusive, thanks to all the denoising going on.

With audio, I think AAC-LC hit critical mass, like H.264, and after that, codecs excel rather in the low-bitrate regime. Opus beats AAC below 128 kbps, but above 160, they converge in quality and compatibility is being traded for no gain. xHE-AAC, on the other hand, seems to span all bitrate domains and soundly beats Opus <= 64 kbps.

In images, we've got a clear winner in JPEG XL, which leaves other formats, lossy and lossless, in the dust and excels at higher quality. Unfortunately, there are forces seemingly trying to undermine its success.

[Z2697]

Originally Posted by GeoffreyA

There is definitely diminishing returns going on in post-H.264 codecs, and they've made filtering detail central to their designs. I think some breakthrough is needed, but fundamentally, there are only so many ways redundancy can be reduced, and throwing away more and more detail becomes the only line of attack. And now, information is increasingly being synthesised on decode, which is seen, for example, in Opus's recent additions.

The Opus additions on decoding are not really synthesising in normal conditions, one synthesizes lost packets and other one tunes the existing post-processing filter for better clarity in extreme low bitrate speech only audio stream.

[birdie]

Originally Posted by ksec

We have shifted to we need higher / best quality at minimal bitrate, instead of having bare minimum / good enough quality at lowest possible bitrate. The two are very different.

I couldn't have said it better.

[GeoffreyA]

Originally Posted by Z2697

The Opus additions on decoding are not really synthesising in normal conditions, one synthesizes lost packets and other one tunes the existing post-processing filter for better clarity in extreme low bitrate speech only audio stream.

Good point.

[birdie]

NVIDIA decided not to support HW VVC decoding in the GeForce 50 series.

That's a major blow to its adoption, which can now easily be postponed for another 2 years. At this point you can say it's just dead for the end consumer.

[kurkosdr]

Originally Posted by birdie

NVIDIA decided not to support HW VVC decoding in the GeForce 50 series.

That's a major blow to its adoption, which can now easily be postponed for another 2 years. At this point you can say it's just dead for the end consumer.

The patent holders holding essential patents for the VVC standard could get together and temporarily waive royalties for all decoder and encoder implementations (software or hardware), so companies like Nvidia and AMD could implement VVC at only silicon cost, but nope, they are like: "we got our IP in an ISO and ITU standard, we deserve a money printer, where the hell is our money printer?".

In fact, the licensing situation for VVC is worse than the one for HEVC, despite VVC being a format used for nothing in most countries. But I guess if the patent holders split themselves into 50 more licensing groups, they will get enough royalties to get their money printer from the 3 companies shipping VVC products, yay! (no, not really)

MPEG has run its course. It's now possible to have a reasonably performant standard without having a ton of patent holders attached to it. Let's remember that MPEG even exists because 35 years ago it was impossible to create a reasonably performant video standard without stepping on someone's patent, but this stopped being true around the time AV1 became a thing.

[GeoffreyA]

This is where greed gets them. There seems little point to VVC, at this point, and I say this as one who championed it in years past.

[Z2697]

Actually I don't think the "big corps" are innocent. If their product is cheap camera that wants a new codec to help promotion but can't afford it due to low margin, maybe that makes some sense.
(some corps holding the patents are "big corps" as well but you get the point)

Of course, I do believe that the codecs should be FREE, but here we are, in a world like this.

[kurkosdr]

Originally Posted by Z2697

Actually I don't think the "big corps" are innocent. If their product is cheap camera that wants a new codec to help promotion but can't afford it due to low margin, maybe that makes some sense.
(some corps holding the patents are "big corps" as well but you get the point)

Generally, codecs are integrated into devices on an "as needed" basis, both due to silicon cost and due to licensing cost. It's the reason DVD players made today still only do MPEG-1, MPEG-2, and MPEG-4 Part 2 (yes, really) because that's what's expected from a DVD player. AVC is considered an "HD format" and not typically supported by DVD players (and no, Sony doesn't care that you can't play your SD H.264 files on their DVD player).

It's the same reason why TVs without HEVC support are still made today (in countries that haven't made the transition to DVB-T2 HEVC yet), because HEVC is a "UHD format" and not expected to be supported by an HDTV.

Which is the problem the VVC patent holders have: nobody needs VVC for any reason (in consumer electronics at least). A handful of very-high-end UHD TVs might have it for future-proofing reasons, but that's it. Really, their only hope at this point is to give it away for free (as far as licensing goes), but of course they won't.

[Z2697]

I suddenly have this idea that a codec (the specification) is free doesn't necessarily mean that it's less profitable.
I mean, the commercial encoding solutions or whatever, and technical support.
Those who don't want to invest money into it, have the choice, unlike royalties, of course.

Just my conspiracy theory perhaps

[birdie]

Too little too late?

https://accessadvance.com/2025/01/16...market-demand/

Access Advance Announces Video Distribution Patent Pool in Response to Market Demand
Pool Covers Internet Streaming of the HEVC, VVC, AV1, and VP9 Video Codecs in a Single License

Responding to growing market demand for an industry solution for codec licensing in the video distribution market, Access Advance LLC (“Advance”) is pleased to announce the launch of its Video Distribution Patent (“VDP”) Pool.

The VDP Pool will build upon the success of Advance’s existing HEVC and VVC Advance Patent Pools, which are supported by a substantial majority of video codec implementers and patent owners. It will provide a single one-stop-shop license, covering internet streaming with all four of the most recently developed video codecs (i.e., HEVC, VVC, AV1, and VP9) available today, with fixed tiered pricing scaled to the size of the video distributor’s business. This structure provides simplicity and predictability to internet video distributors, allowing them to choose which codec(s) to use based on technical and business merits rather than royalty costs or the need to negotiate multiple licenses.

[kurkosdr]

Originally Posted by birdie

Too little too late?

https://accessadvance.com/2025/01/16...market-demand/

This is just for video distribution (aka for "content fees"), it doesn't concern vendors of encoders or decoders (so it doesn't concern implementers of encoders and decoders like Nvidia), and it's just one patent licensing entity of the many for VVC. Also, we don't know pricing.

The interesting bit is that Access Advance thinks they have patents essential for VP9 and AV1 video streaming in this patent pool. Will the patent holders go after Google (YouTube) for "content fees"? The patent holders in the Sisvel pools haven't gone after anyone despite their VP9 and AV1 patent pools being several years old by now, but I guess I am in the wrong thread for that (edit: created a thread)

[benwaggoner]

Originally Posted by kurkosdr

This is just for video distribution (aka for "content fees"), it doesn't concern vendors of encoders or decoders (so it doesn't concern implementers of encoders and decoders like Nvidia), and it's just one patent licensing entity of the many for VVC. Also, we don't know pricing.

Ambiguity in "content fees" is what can kill the encoder and decoder businesses though, as it can prevent demand for either.

[kurkosdr]

Originally Posted by benwaggoner

Ambiguity in "content fees" is what can kill the encoder and decoder businesses though, as it can prevent demand for either.

And that ambiguity is still there for VVC, Access Advance is merely 1 out of the 20 patent licensing entities for VVC. Each of the remaining 19 has the right to demand its own distribution royalty ("content fee").

Generally, when it comes to streaming, I expect HEVC to be the end of the line for "FRAND" standards designed to step on as many patents as possible to achieve marginal gains (which is what the ISO and ITU video standards essentially are) and the future of streaming belongs to standards designed to step on zero patents if possible (excluding patents made available on a royalty-free basis). Internet speeds are getting better every year, so the streaming industry doesn't see the need to negotiate with 20 different licensing entities for marginal compression performance gains. I am even willing to bet Netflix could walk back HEVC if they could (they can't for back compat reasons) and VVC has even more greed around it than HEVC.

[benwaggoner]

Originally Posted by kurkosdr

And that ambiguity is still there for VVC, Access Advance is merely 1 out of the 20 patent licensing entities for VVC. Each of the remaining 19 has the right to demand its own distribution royalty ("content fee").

Generally, when it comes to streaming, I expect HEVC to be the end of the line for "FRAND" standards designed to step on as many patents as possible to achieve marginal gains (which is what the ISO and ITU video standards essentially are) and the future of streaming belongs to standards designed to step on zero patents if possible (excluding patents made available on a royalty-free basis). Internet speeds are getting better every year, so the streaming industry doesn't see the need to negotiate with 20 different licensing entities for marginal compression performance gains. I am even willing to bet Netflix could walk back HEVC if they could (they can't for back compat reasons) and VVC has even more greed around it than HEVC.

Well, there's no arguing that MPEG is still making really good codecs. VVC offers great compression efficiency gains with relatively minor decode complexity increases. It certainly offers better efficiency than AV1 with lower decoder complexity. They've been pretty disciplined about only allowing in tools that offer worthwhile bitrate savings for the complexity.

But yeah, entirely decoupling technical design from IP questions has, in my personal opinion, failed for a couple of generations in a row now, and I don't see how taking the same approach for H.267 wouldn't yield the same problems.

And the classic MPEG approach failed with MPEG-4 part 2 as well. It was really only with MPEG-2 and AVC/H.264 where we wound up with a single patent pool with reasonable published terms that companies felt reasonably safe with (although there certainly was patent litigation anyway). The failure of Pt. 2 and the competition from VC-1 seemed to get H.264 patent holders scared enough to work together. I'm not sure why the threat of AV1/AV2 isn't doing the same thing today. The explosion of non-practicing entities resulting in a lot of IP is owned by companies without any stake in the ecosystem beyond rent extraction seems a likely factor.

I'm very glad to be a technologist with personal opinions, not someone whose day job is dealing with this IP stuff!