Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 24th May 2020, 17:51   #1  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,464
x265 buffer underflows

Getting buffer underflow errors when encoding CBR with x265 using strict-cbr. Is it a known issue?
Another issue could be fact that I'm forcing I frames with ffmpeg's -force_key_frames "expr:eq(mod(n,50),0)" This may be messing with rate control.
Is there any other way of getting fixed 2 seconds IDR frame distance (and still letting encoder to use I frames inside those 2 seconds)? Using latest ffmpeg.

Last edited by kolak; 24th May 2020 at 18:26.
kolak is offline   Reply With Quote
Old 24th May 2020, 18:51   #2  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 4,219
Quote:
Originally Posted by kolak View Post
Getting buffer underflow errors when encoding CBR with x265 using strict-cbr. Is it a known issue?
Another issue could be fact that I'm forcing I frames with ffmpeg's -force_key_frames "expr:eq(mod(n,50),0)" This may be messing with rate control.
Is there any other way of getting fixed 2 seconds IDR frame distance (and still letting encoder to use I frames inside those 2 seconds)? Using latest ffmpeg.
Did you try -g 50 ?

Or -x265-params keyint=50
poisondeathray is offline   Reply With Quote
Old 24th May 2020, 19:31   #3  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,464
g or keying itself means max GOP size. It doesn't guarantee you IDR frame at every n*50fps at all (only that no GOP will be bigger than 50).
I need IDR frame at ever n*50 + I frames (to cover scene changes etc.) in between if encoder decides so.

Last edited by kolak; 24th May 2020 at 20:25.
kolak is offline   Reply With Quote
Old 24th May 2020, 19:36   #4  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,487
Quote:
Originally Posted by kolak View Post
g or keying itself means max GOP size. It doesn't guarantee you IDR frame at every n*50fps at all (only fact that no GOP will be bigger than 50).
I need IDR frame at ever n*50 + I frames (to cover scene changes etc.) in between if encoders decides so.
use multipass perhaps. x265 has problem for size RC estimation.
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 24th May 2020, 20:25   #5  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,464
Looks like 2 pass may be the only option.
Problem is that it cost so much extra time. I don't necessarily have this luxury to waste 2x more time.
kolak is offline   Reply With Quote
Old 24th May 2020, 21:36   #6  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,696
Quote:
Is there any other way of getting fixed 2 seconds IDR frame distance (and still letting encoder to use I frames inside those 2 seconds)
Yep, just set keyint == min-keyint and it will work fine inserting I frames (not IDR) on scene changes inside your GOP.
Blue_MiSfit is offline   Reply With Quote
Old 24th May 2020, 22:29   #7  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,487
Quote:
Originally Posted by kolak View Post
Looks like 2 pass may be the only option.
Problem is that it cost so much extra time. I don't necessarily have this luxury to waste 2x more time.
well try that:

Quote:
x265.exe --input C:\ToS_1920x800_xdither.y4m --output ToS-2000-psy.265 --input-res 1920x800 --fps 24000/1000 --bitrate 2000 --strict-cbr --vbv-maxrate 2000 --vbv-bufsize 12000 --pass 1 --slow-firstpass --stats ToS-2000-psy.log --preset veryslow --rd-refine --min-keyint 1 --keyint 120 --ipratio 1.00 --pbratio 1.00 --analysis-save ToS_1000_cbr_analysis.dat --analysis-save-reuse-level 10

x265.exe --input C:\ToS_1920x800_xdither.y4m --output ToS-2000-psy.265 --input-res 1920x800 --fps 24000/1000 --bitrate 2000 --strict-cbr --vbv-maxrate 2000 --vbv-bufsize 12000 --pass 3 --slow-firstpass --stats ToS-2000-psy.log --preset veryslow --rd-refine --min-keyint 1 --keyint 120 --ipratio 1.00 --pbratio 1.00 --analysis-load ToS_1000_cbr_analysis.dat --analysis-load-reuse-level 10

x265.exe --input C:\ToS_1920x800_xdither.y4m --output ToS-2000-psy.265 --input-res 1920x800 --fps 24000/1000 --bitrate 2000 --strict-cbr --vbv-maxrate 2000 --vbv-bufsize 12000 --pass 3 --slow-firstpass --stats ToS-2000-psy.log --preset veryslow --rd-refine --min-keyint 1 --keyint 120 --ipratio 1.00 --pbratio 1.00 --analysis-load ToS_1000_cbr_analysis.dat --analysis-load-reuse-level 10

..... etc etc

Reuse encoding use complete research from first pass and refine Rate Control at really high speed (like fastest preset speed but with veryslow preset quality). And reuse mode will refine and refine and refine again Rate Control in Npass mode certainely with better quality than classic 2 pass RC mode, and with better constrained Rate Control at each pass.

IMO with this particular mode encoding, --strict-cbr will be even useless if you use many pass and you can expect better quality too.
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 24th May 2020, 22:38   #8  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,487
and you can certainely expect more constant overall quality and less underflow with something like that:

--bitrate 1900 --vbv-maxrate 2000 --vbv-bufsize 12000

if your target bitrate is 2000, choose little lower bitrate like 1900 to have better buffer gestion and more constant quality in GOP between I, P and Bframe.
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 24th May 2020, 23:03   #9  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,464
How is this then CBR encoding?
My spec says CBR at 2% tolerance which is bit crazy
kolak is offline   Reply With Quote
Old 24th May 2020, 23:06   #10  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,487
Quote:
Originally Posted by kolak View Post
How is this then CBR encoding?
My spec says CBR at 2% tolerance which is bit crazy
certainely usefull for max bitrate limit, but why for min bitrate limit?
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 24th May 2020, 23:09   #11  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,464
Because spec is set by someone who has not much clue about encoding, I assume
kolak is offline   Reply With Quote
Old 24th May 2020, 23:13   #12  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,464
Quote:
Originally Posted by Blue_MiSfit View Post
Yep, just set keyint == min-keyint and it will work fine inserting I frames (not IDR) on scene changes inside your GOP.
This is what I thought but had strange results. Need to check it again as it was possibly due to reading some bad first pass details left after some testing
kolak is offline   Reply With Quote
Old 24th May 2020, 23:14   #13  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,487
Quote:
Originally Posted by kolak View Post
Because spec is set by someone who has not much clue about encoding, I assume
Well if you have multiple consecutive pure black frame, encoder will certainely never use high bitrate for that ... ;-)
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 24th May 2020, 23:15   #14  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,464
Yes, this is very possible as content changes a lot and has many dips to blacks and overall is very dark.
Is there a solution for this?
kolak is offline   Reply With Quote
Old 24th May 2020, 23:21   #15  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,464
Quote:
Originally Posted by kolak View Post
This is what I thought but had strange results. Need to check it again as it was possibly due to reading some bad first pass details left after some testing

Strange getting GOPs at 61, 72 ... for keyint=minkeyint=50
(update: it was due to gop-lookahead not been set to 0)

I'm actually not getting underflow errors anymore (at least not so far), so looks like ffmpeg's force_keyframe option messes a bit with rate control in x265.

Last edited by kolak; 26th May 2020 at 16:40.
kolak is offline   Reply With Quote
Old 1st June 2020, 19:52   #16  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,242
Quote:
Originally Posted by kolak View Post
Strange getting GOPs at 61, 72 ... for keyint=minkeyint=50
(update: it was due to gop-lookahead not been set to 0)
What was your scenario for "gop-lookahead?" I've never figured out what the feature was actually for, versus just changing min-keyint directly.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 2nd June 2020, 15:08   #17  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,464
I did not understand it either, now I think I do.
Let's say you have keyint set to 50 frames. If you set gop-lookahead to eg. 25 frames, then x265 will look those 25 further ahead to see if it may be better to set an IDR frame somewhere there. It may decide that best IDR place will be eg. 10 frames after those 50, so you will end up with 60 frames GOP.
If you do the same, but set gop-lookahead=0 at those 10 frames pass 50 frames x265 will most likely set I frame (not an IDR). This will properly respect 50 frames maximum GOP.
With gop-lookahead your longest GOP is keyint+gop-lookahead.
I may be wrong
No idea how it's different than setting keyint=75

Quote:
--gop-lookahead <integer>
Number of frames for GOP boundary decision lookahead. If a scenecut frame is found within this from the gop boundary set by –keyint, the GOP will be extented until such a point, otherwise the GOP will be terminated as set by –keyint. Default 0.

Range of values: Between 0 and (–rc-lookahead - mini-GOP length)

It is recommended to have –gop-lookahaed less than –min-keyint as scenecuts beyond –min-keyint are already being coded as keyframes.

Last edited by kolak; 2nd June 2020 at 15:14.
kolak is offline   Reply With Quote
Old 2nd June 2020, 20:12   #18  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,696
Ah, so if you do want constant IDR cadence (like for ABR streaming) then you would presumably want to make sure that's disabled! (it is by default)

Seems like a great feature for a satellite / broadcast feed where the main constraint on keyint is latency and channel switch time. Having slight wiggle room is probably acceptable in those cases.
Blue_MiSfit is offline   Reply With Quote
Old 4th June 2020, 20:34   #19  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,242
Quote:
Originally Posted by Blue_MiSfit View Post
Ah, so if you do want constant IDR cadence (like for ABR streaming) then you would presumably want to make sure that's disabled! (it is by default)

Seems like a great feature for a satellite / broadcast feed where the main constraint on keyint is latency and channel switch time. Having slight wiggle room is probably acceptable in those cases.
So, --keint 50 --gop-lookahead 25 would still have a max GOP duration of 50 frames, but the average would tend to be closer to 50 than 75?

Seems worthy of some testing.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 4th June 2020, 23:18   #20  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,696
Quote:
--keint 50 --gop-lookahead 25
My interpretation is that the nominal GOP length would be 50, but that it might be as high as 75 in some cases, if a scenecut is found within that extra 25 frames.

Put another way, starting at frame 0, assuming scene change detection is not triggered from frames 0-49, frame 50 will be an IDR... UNLESS frames 51-74 are flagged by scene change detection, in which case the earliest of those will be the IDR.

At least.. I think I did the math right there I always have to think twice about duration vs frame number etc!

Last edited by Blue_MiSfit; 4th June 2020 at 23:24.
Blue_MiSfit is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 05:57.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.