Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 22nd April 2020, 17:08   #1  |  Link
excellentswordfight
Lost my old account :(
 
Join Date: Jul 2017
Posts: 149
CRF use for content aware VOD streaming?

Hi,

So there are a lot of discussions regarding content aware/per-title streaming for VOD nowdays, were different types of methods are used to select what bitrate is appropriate when it comes to streaming.

old, but yeah basically this:
https://netflixtechblog.com/per-titl...n-7e99442b62a2

But it seems like a lot of streaming services (i guess at least most of the smaller local ones) still just use fixed bitrates for the ladder, cause I guess they dont have resources to implement complex workflows.

So I've been wondering, why isnt CRF used (or is it?)? Cant it be in used together with vbv restrictions to create a crude method to do this?

I'm a bit out of my realm of expertise here but. Lets say you have a 1Mbps connection target preset, why not set a crf value that is close to 500Kbps in avg bitrate for your average complexity material, then set --vbv-maxrate 750 --vbv-bufsize 1000. I'm I missing something or shouldn't this be a fairly good (or at least easy) way of doing it compared to fixed bitrates?

So a ladder would look something like this:

540p24 ~500Kbps (1Mbps safe) --crf 26 --vbv-maxrate 750 --vbv-bufsize 1000

720p24 ~1250Kbps (2Mbps safe) --crf 24 --vbv-maxrate 1750 --vbv-bufsize 2500

1080p24 ~3Mbps (5Mbps safe) --crf 22 --vbv-maxrate 4500 --vbv-bufsize 6000

1080p24 ~6Mbps (10Mbps safe) --crf 19 --vbv-maxrate 9000 --vbv-bufsize 12000

So, my question is basically: is this a good or a poor way of doing it? If good, is this already in widespread use, and if not, why?

Last edited by excellentswordfight; 22nd April 2020 at 17:28.
excellentswordfight is offline   Reply With Quote
Old 22nd April 2020, 19:13   #2  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,732
This is a reasonable way of doing things. Some providers do exactly this.

Beamr did an interesting review of the landscape and found several content adaptation methods

http://media2.beamrvideo.com/pdf/Bea...Tech_Guide.pdf

Beamr's tech is cool, they do frame by frame optimization in loop with encoding, so they calculate a lot of metrics for an encoded frame and then re-encode the frame with fewer and fewer bits until they get the lowest data rate possible while maintaining the metric score. It's more complex than that, but it's a tight integration. I've looked at their stuff and it's quite good. It's not cheap tho

Netflix does extensive content optimization (the per scene convex hull approach).
Blue_MiSfit is offline   Reply With Quote
Old 22nd April 2020, 19:39   #3  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,357
Yeah, using CRF can definitely be a useful tool in this regard. One problem is that some older heuristics struggle with the variable bitrate, especially if they don't make an internal distinction between peak bitrate and actual bitrate. So you'll see this sort of technique used more by companies who either focus only on web or recent devices, or who are able to deliver their own player with custom heuristics. If one is trying to work with the build-in DASH client in older living room devices, things get fraught quickly.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 22nd April 2020, 20:15   #4  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,357
Looks like someone at Amazon patented some related ideas . http://pat2pdf.org/patents/pat9712860.pdf
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 22nd April 2020, 20:25   #5  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,732
Makes sense. Smart TVs in general (especially older ones) are _ATROCIOUS_ platforms to develop for, especially when you have to live in the sandbox environment they give app developers. Netflix always gets special permission to run their own binary. Nobody else gets that level of integration, so it limits what's possible.
Blue_MiSfit is offline   Reply With Quote
Old 24th April 2020, 00:01   #6  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,357
Quote:
Originally Posted by Blue_MiSfit View Post
Makes sense. Smart TVs in general (especially older ones) are _ATROCIOUS_ platforms to develop for, especially when you have to live in the sandbox environment they give app developers. Netflix always gets special permission to run their own binary. Nobody else gets that level of integration, so it limits what's possible.
I wouldn't say nobody else. I'd expect it is available to the big five premium content services. They build players in a lot of different ways targeting different levels of abstraction.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 24th April 2020, 00:25   #7  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,732
That's true, I guess having worked for one of the big studios and still running into these limitations I assumed that would be the case for other platforms as well
Blue_MiSfit is offline   Reply With Quote
Old 24th April 2020, 16:36   #8  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,357
Quote:
Originally Posted by Blue_MiSfit View Post
That's true, I guess having worked for one of the big studios and still running into these limitations I assumed that would be the case for other platforms as well
Other than Disney+, the big studios's official platforms don't have a significant market share. Here's a recent breakdown: https://www.mediapost.com/publicatio...=factoftheday1

Because only Disney has core brands customers are familiar with (Disney, Marvel, Pixar). People just don't know which shows and movies would be on Peacock versus CBS All Access.

Building and supporting premimum experience video players for all the devices people want to watch on (so many generations of OS software running on so many different SoCs!) and the content for those (different codecs, packaging, and DRM systems) takes a 1000+ person engineering org by my estimate.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 26th April 2020, 22:27   #9  |  Link
TEB
Registered User
 
Join Date: Feb 2003
Location: Palmcoast of Norway
Posts: 348
We are actually doing this as we speak.
We dont have need to go as far as netflix/youtube are going. So we went the CRF + CAP way with a fixed ladder. Gives quite results as we are only allowing newer devices on native players..
TEB is offline   Reply With Quote
Old 27th April 2020, 23:20   #10  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,357
Quote:
Originally Posted by TEB View Post
We are actually doing this as we speak.
We dont have need to go as far as netflix/youtube are going. So we went the CRF + CAP way with a fixed ladder. Gives quite results as we are only allowing newer devices on native players..
By CAP do you mean capped VBV?

It is a delicious luxury to only have to worry about newer devices! Of course, the longer a service lasts, the more today's bright and shiny will eventually become tomorrow legacy deadweight holding you back .
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 28th April 2020, 09:39   #11  |  Link
TEB
Registered User
 
Join Date: Feb 2003
Location: Palmcoast of Norway
Posts: 348
Quote:
Originally Posted by benwaggoner View Post
By CAP do you mean capped VBV?

It is a delicious luxury to only have to worry about newer devices! Of course, the longer a service lasts, the more today's bright and shiny will eventually become tomorrow legacy deadweight holding you back .
X264:
-crf 20 -maxrate 15M -bufsize 15M for the top profile
TEB is offline   Reply With Quote
Old 28th April 2020, 09:48   #12  |  Link
TEB
Registered User
 
Join Date: Feb 2003
Location: Palmcoast of Norway
Posts: 348
So for us, we (ISP and Operator) finally got to the point that we had the opportunity to move from a classic static CBR ladder til something better.

We had to decide on how far we wanted to take this into the constant quality domain and we ended up with the conclusion that we wanted to do something between what we have today and what Netflix is doing!

We did a lot of tests of 2pass VBR, vs CRF "free" vs CRF Capped, vs CBR both on x264, x265 and commercial encoders like Titan File

We ended up with a static ladder based on CRF with a bitrate CAP which gives us a lot of what Netflix is doing, but not going all the way with regards to scene based encoding, title tuning and multiple pre-passes to find the sweet spot etc..

To give an example:
Our top HQ 1080p profile/rung is basically CRF19@15mbps CAP with a fallback of CRF22@8mbps CAP etc..
TEB is offline   Reply With Quote
Old 28th April 2020, 18:56   #13  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,357
Quote:
Originally Posted by TEB View Post
X264:
-crf 20 -maxrate 15M -bufsize 15M for the top profile
Oh, if that was the only hard part! But what about the rare cases where the first NAL unit is larger than the following, which is allowed by the spec, but not by some SoC decoders? Or the many, many permutations of DRM encryption that can be fussy with some devices. Or Qualcomm DRM carveouts where you need to allocate on boot enough memory for the maximum resolution & max reference frames. Note that it's the worst case of both; supporting 320x180 with 6 reference frames and 3840x2160 with 3 reference frames requires 3840x2160x6 memory carveout.

And then there are all the heuristics that will treat a 15 Mbps peakrate as all fragments of that stream index being really 15 Mbps, so it won't go to that even when that top stream is at 3 Mbps and there's 12 Mbps of bandwidth.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 28th April 2020, 18:58   #14  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,357
Quote:
Originally Posted by TEB View Post
So for us, we (ISP and Operator) finally got to the point that we had the opportunity to move from a classic static CBR ladder til something better.

We had to decide on how far we wanted to take this into the constant quality domain and we ended up with the conclusion that we wanted to do something between what we have today and what Netflix is doing!
What's the stuff Netflix is doing that you aren't?

Quote:
We ended up with a static ladder based on CRF with a bitrate CAP which gives us a lot of what Netflix is doing, but not going all the way with regards to scene based encoding, title tuning and multiple pre-passes to find the sweet spot etc..

To give an example:
Our top HQ 1080p profile/rung is basically CRF19@15mbps CAP with a fallback of CRF22@8mbps CAP etc..
Do you have documentation on what Netflix is doing with Capped VBR (CVBR)?

Note that CRF itself IS scene-based encoding. There are other flavors of it, but the most basic is not to spend bits where they aren't needed.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 29th April 2020, 10:06   #15  |  Link
excellentswordfight
Lost my old account :(
 
Join Date: Jul 2017
Posts: 149
Quote:
Originally Posted by benwaggoner View Post
Oh, if that was the only hard part! But what about the rare cases where the first NAL unit is larger than the following, which is allowed by the spec, but not by some SoC decoders? Or the many, many permutations of DRM encryption that can be fussy with some devices. Or Qualcomm DRM carveouts where you need to allocate on boot enough memory for the maximum resolution & max reference frames. Note that it's the worst case of both; supporting 320x180 with 6 reference frames and 3840x2160 with 3 reference frames requires 3840x2160x6 memory carveout.
I dont really follow how this can be an issue. Isnt the decoder validated for levels and profiles? If a decoder is specified to handle main10 main teir level 4, how could it not handle MaxDpbSize and vbv-limits within the spec? As long as the buffer is not higher then the maximum maxrate of that level/tier and that it can feed data in that rate shoudlnt it be all good? Or is this specifically an issue for some DRM solutions?
Quote:
And then there are all the heuristics that will treat a 15 Mbps peakrate as all fragments of that stream index being really 15 Mbps, so it won't go to that even when that top stream is at 3 Mbps and there's 12 Mbps of bandwidth.
Mind expanding on this? Why would it use the peakrate as decision for selection this way? Shouldn't the decision be made on buffer status and chunk sizes? Would that really be an issue with ABR with MPEG-DASH and HLS?

Last edited by excellentswordfight; 29th April 2020 at 10:59.
excellentswordfight is offline   Reply With Quote
Old 29th April 2020, 20:03   #16  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,732
In my experience, the DASH clients on older devices are insane. If those devices are important enough, then you need to make playback work well on them. It kind of doesn't matter what the problem really is.

This can ALSO be an issue with poorly implemented DRM, btw. I've definitely seen certain popular settop boxes blow up when fed streams that were 100% valid within the constraints of the Main10 profile @ level 5 (both vbv limits and MaxDpbSize). Reducing the reference frame count was the only solution. This was acknowledged as a firmware bug but I don't know if it's ever been fixed
Blue_MiSfit is offline   Reply With Quote
Old 30th April 2020, 09:37   #17  |  Link
TEB
Registered User
 
Join Date: Feb 2003
Location: Palmcoast of Norway
Posts: 348
Quote:
Originally Posted by benwaggoner View Post
What's the stuff Netflix is doing that you aren't?
Do you have documentation on what Netflix is doing with Capped VBR (CVBR)?
Note that CRF itself IS scene-based encoding. There are other flavors of it, but the most basic is not to spend bits where they aren't needed.
I merely pointing at what i have read through their excellent blogs as well as general knowledge in this marked (i may be totally wrong on my assumptions, but thats why we are all here to learn ):
1. Using some kind of ML process to detect the scenes, separation
2. Encoding each scene seperatly with different encoding parameters (sweet spot), maybe different gop lengths also??
3. Use loops of CRF tests against a VMAF threshold to find the baseline for encoding with static/capped VBR.
TEB is offline   Reply With Quote
Old 30th April 2020, 09:42   #18  |  Link
TEB
Registered User
 
Join Date: Feb 2003
Location: Palmcoast of Norway
Posts: 348
Quote:
Originally Posted by Blue_MiSfit View Post
In my experience, the DASH clients on older devices are insane. If those devices are important enough, then you need to make playback work well on them. It kind of doesn't matter what the problem really is.

This can ALSO be an issue with poorly implemented DRM, btw. I've definitely seen certain popular settop boxes blow up when fed streams that were 100% valid within the constraints of the Main10 profile @ level 5 (both vbv limits and MaxDpbSize). Reducing the reference frame count was the only solution. This was acknowledged as a firmware bug but I don't know if it's ever been fixed
I hear ya Luckily we have a higher minimum device req.
With that said, what we DO have seen as the major issue is actually Lvl1 WV in the combination with 1080p50 for linear sports broadcasts.
Alot of embedded devices running AOSP/ATV cant properly do this without dropping frames in the TEE environment..
Apple IOS/APPLE eats most of our content with little complaint as long as we have ios10+ as the minimum os level req
TEB is offline   Reply With Quote
Old 30th April 2020, 09:43   #19  |  Link
TEB
Registered User
 
Join Date: Feb 2003
Location: Palmcoast of Norway
Posts: 348
Quote:
Originally Posted by benwaggoner View Post
Oh, if that was the only hard part! But what about the rare cases where the first NAL unit is larger than the following, which is allowed by the spec, but not by some SoC decoders? Or the many, many permutations of DRM encryption that can be fussy with some devices. Or Qualcomm DRM carveouts where you need to allocate on boot enough memory for the maximum resolution & max reference frames. Note that it's the worst case of both; supporting 320x180 with 6 reference frames and 3840x2160 with 3 reference frames requires 3840x2160x6 memory carveout.

And then there are all the heuristics that will treat a 15 Mbps peakrate as all fragments of that stream index being really 15 Mbps, so it won't go to that even when that top stream is at 3 Mbps and there's 12 Mbps of bandwidth.
Woa!! Thats alot of corner cases i have never seen/observed.
We dont go past 1080p50 today and have relatively modern devices mentioned in other replies.
TEB is offline   Reply With Quote
Old 1st May 2020, 00:10   #20  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,732
Quote:
Originally Posted by TEB View Post
I hear ya Luckily we have a higher minimum device req.
With that said, what we DO have seen as the major issue is actually Lvl1 WV in the combination with 1080p50 for linear sports broadcasts.
Alot of embedded devices running AOSP/ATV cant properly do this without dropping frames in the TEE environment..
Apple IOS/APPLE eats most of our content with little complaint as long as we have ios10+ as the minimum os level req
Yeah, throughput for AES-CTR mode becomes a major issue on cost constrained devices. Apple was right to support only AES-CBCS since it' so much faster. Incidentally they were right for the wrong reasons Their devices are beefy enough to do CTR at high speeds! We've seen on certain cheap crap Smart TVs that support both formats that switching from CTR to CBCS almost doubled throughput.

Anyway, we're drifting off-topic here a bit, even if this is still relevant in the context of the risks associated with not using more traditional CBR / heavily capped VBR rate control modes for streaming
Blue_MiSfit is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 18:08.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.