CRF use for content aware VOD streaming?

[excellentswordfight]

Hi,

So there are a lot of discussions regarding content aware/per-title streaming for VOD nowdays, were different types of methods are used to select what bitrate is appropriate when it comes to streaming.

old, but yeah basically this:
https://netflixtechblog.com/per-titl...n-7e99442b62a2

But it seems like a lot of streaming services (i guess at least most of the smaller local ones) still just use fixed bitrates for the ladder, cause I guess they dont have resources to implement complex workflows.

So I've been wondering, why isnt CRF used (or is it?)? Cant it be in used together with vbv restrictions to create a crude method to do this?

I'm a bit out of my realm of expertise here but. Lets say you have a 1Mbps connection target preset, why not set a crf value that is close to 500Kbps in avg bitrate for your average complexity material, then set --vbv-maxrate 750 --vbv-bufsize 1000. I'm I missing something or shouldn't this be a fairly good (or at least easy) way of doing it compared to fixed bitrates?

So a ladder would look something like this:

540p24 ~500Kbps (1Mbps safe) --crf 26 --vbv-maxrate 750 --vbv-bufsize 1000

720p24 ~1250Kbps (2Mbps safe) --crf 24 --vbv-maxrate 1750 --vbv-bufsize 2500

1080p24 ~3Mbps (5Mbps safe) --crf 22 --vbv-maxrate 4500 --vbv-bufsize 6000

1080p24 ~6Mbps (10Mbps safe) --crf 19 --vbv-maxrate 9000 --vbv-bufsize 12000

So, my question is basically: is this a good or a poor way of doing it? If good, is this already in widespread use, and if not, why?

[Blue_MiSfit]

This is a reasonable way of doing things. Some providers do exactly this.

Beamr did an interesting review of the landscape and found several content adaptation methods

http://media2.beamrvideo.com/pdf/Bea...Tech_Guide.pdf

Beamr's tech is cool, they do frame by frame optimization in loop with encoding, so they calculate a lot of metrics for an encoded frame and then re-encode the frame with fewer and fewer bits until they get the lowest data rate possible while maintaining the metric score. It's more complex than that, but it's a tight integration. I've looked at their stuff and it's quite good. It's not cheap tho

Netflix does extensive content optimization (the per scene convex hull approach).

[benwaggoner]

Yeah, using CRF can definitely be a useful tool in this regard. One problem is that some older heuristics struggle with the variable bitrate, especially if they don't make an internal distinction between peak bitrate and actual bitrate. So you'll see this sort of technique used more by companies who either focus only on web or recent devices, or who are able to deliver their own player with custom heuristics. If one is trying to work with the build-in DASH client in older living room devices, things get fraught quickly.

[benwaggoner]

Looks like someone at Amazon patented some related ideas . http://pat2pdf.org/patents/pat9712860.pdf

[Blue_MiSfit]

Makes sense. Smart TVs in general (especially older ones) are _ATROCIOUS_ platforms to develop for, especially when you have to live in the sandbox environment they give app developers. Netflix always gets special permission to run their own binary. Nobody else gets that level of integration, so it limits what's possible.

[benwaggoner]

Originally Posted by Blue_MiSfit

Makes sense. Smart TVs in general (especially older ones) are _ATROCIOUS_ platforms to develop for, especially when you have to live in the sandbox environment they give app developers. Netflix always gets special permission to run their own binary. Nobody else gets that level of integration, so it limits what's possible.

I wouldn't say nobody else. I'd expect it is available to the big five premium content services. They build players in a lot of different ways targeting different levels of abstraction.

[Blue_MiSfit]

That's true, I guess having worked for one of the big studios and still running into these limitations I assumed that would be the case for other platforms as well

[benwaggoner]

Originally Posted by Blue_MiSfit

That's true, I guess having worked for one of the big studios and still running into these limitations I assumed that would be the case for other platforms as well

Other than Disney+, the big studios's official platforms don't have a significant market share. Here's a recent breakdown: https://www.mediapost.com/publicatio...=factoftheday1

Because only Disney has core brands customers are familiar with (Disney, Marvel, Pixar). People just don't know which shows and movies would be on Peacock versus CBS All Access.

Building and supporting premimum experience video players for all the devices people want to watch on (so many generations of OS software running on so many different SoCs!) and the content for those (different codecs, packaging, and DRM systems) takes a 1000+ person engineering org by my estimate.

[TEB]

We are actually doing this as we speak.
We dont have need to go as far as netflix/youtube are going. So we went the CRF + CAP way with a fixed ladder. Gives quite results as we are only allowing newer devices on native players..

[benwaggoner]

Originally Posted by TEB

We are actually doing this as we speak.
We dont have need to go as far as netflix/youtube are going. So we went the CRF + CAP way with a fixed ladder. Gives quite results as we are only allowing newer devices on native players..

By CAP do you mean capped VBV?

It is a delicious luxury to only have to worry about newer devices! Of course, the longer a service lasts, the more today's bright and shiny will eventually become tomorrow legacy deadweight holding you back .

[TEB]

Originally Posted by benwaggoner

By CAP do you mean capped VBV?

It is a delicious luxury to only have to worry about newer devices! Of course, the longer a service lasts, the more today's bright and shiny will eventually become tomorrow legacy deadweight holding you back .

X264:
-crf 20 -maxrate 15M -bufsize 15M for the top profile

[Blue_MiSfit]

In my experience, the DASH clients on older devices are insane. If those devices are important enough, then you need to make playback work well on them. It kind of doesn't matter what the problem really is.

This can ALSO be an issue with poorly implemented DRM, btw. I've definitely seen certain popular settop boxes blow up when fed streams that were 100% valid within the constraints of the Main10 profile @ level 5 (both vbv limits and MaxDpbSize). Reducing the reference frame count was the only solution. This was acknowledged as a firmware bug but I don't know if it's ever been fixed

[TEB]

Originally Posted by Blue_MiSfit

In my experience, the DASH clients on older devices are insane. If those devices are important enough, then you need to make playback work well on them. It kind of doesn't matter what the problem really is.

This can ALSO be an issue with poorly implemented DRM, btw. I've definitely seen certain popular settop boxes blow up when fed streams that were 100% valid within the constraints of the Main10 profile @ level 5 (both vbv limits and MaxDpbSize). Reducing the reference frame count was the only solution. This was acknowledged as a firmware bug but I don't know if it's ever been fixed

I hear ya Luckily we have a higher minimum device req.
With that said, what we DO have seen as the major issue is actually Lvl1 WV in the combination with 1080p50 for linear sports broadcasts.
Alot of embedded devices running AOSP/ATV cant properly do this without dropping frames in the TEE environment..
Apple IOS/APPLE eats most of our content with little complaint as long as we have ios10+ as the minimum os level req

[Blue_MiSfit]

Originally Posted by TEB

I hear ya Luckily we have a higher minimum device req.
With that said, what we DO have seen as the major issue is actually Lvl1 WV in the combination with 1080p50 for linear sports broadcasts.
Alot of embedded devices running AOSP/ATV cant properly do this without dropping frames in the TEE environment..
Apple IOS/APPLE eats most of our content with little complaint as long as we have ios10+ as the minimum os level req

Yeah, throughput for AES-CTR mode becomes a major issue on cost constrained devices. Apple was right to support only AES-CBCS since it' so much faster. Incidentally they were right for the wrong reasons Their devices are beefy enough to do CTR at high speeds! We've seen on certain cheap crap Smart TVs that support both formats that switching from CTR to CBCS almost doubled throughput.

Anyway, we're drifting off-topic here a bit, even if this is still relevant in the context of the risks associated with not using more traditional CBR / heavily capped VBR rate control modes for streaming