Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 19th August 2018, 13:40   #21  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,308
I may have an idea... Are you able to provide a source which trig the issue creating a file like this :
You choose a small part where the scene change trig the issue, the sample will have 3 scenes, so 2 change scene :
[Scene 1][Scene 2][Scene 3]. The issue is trigged between scene 2 and Scene 3.
You create an avs script like this (i'm assuming input file is an avi file in lossless codec) :
Code:
a = AviSource("Small_Sample.avi").trim(0,xxxx)
b = AviSource("Small_Sample.avi").trim(xxxx+1,0)
a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+b
xxxxx/xxxx+1 : The frame number of the scene change between Scene 1 and Scene 2. So a will have only [Scene 1], and b will have [Scen2][Scene3}.
Put enough "a" to finaly be able to trig the issue at the end of the file.
Will that trick enable you to provide a trigged sample, without having to provode the whole file (which is of course totaly not possible) ?
__________________
My github.
jpsdr is offline   Reply With Quote
Old 19th August 2018, 17:15   #22  |  Link
mp3dom
Registered User
 
Join Date: Jul 2003
Location: Italy
Posts: 1,135
Ok, found a good trim.
If some dev really want to look into that, please drop me a PM (I can't share the source publicly).
Thanks.

Last edited by mp3dom; 19th August 2018 at 23:10.
mp3dom is offline   Reply With Quote
Old 21st August 2018, 10:15   #23  |  Link
Sharc
Registered User
 
Join Date: May 2006
Posts: 3,997
In case something positive will develop it would be great to learn about it here. Thanks.
Sharc is offline   Reply With Quote
Old 21st August 2018, 13:32   #24  |  Link
mp3dom
Registered User
 
Join Date: Jul 2003
Location: Italy
Posts: 1,135
Yep, so far the problem appears even with threads=1, and with ABR the problem is even more evident (it last 4 seconds, while with CRF or 2-pass it last up to 1 second).
I gave the source to one dev anyway, so if there's something that needs to be fixed, I guess the git will be updated.
Thanks.
mp3dom is offline   Reply With Quote
Old 21st August 2018, 14:24   #25  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,308
Ok, at least, things are on their way. After, it will be done when it will be done...
__________________
My github.
jpsdr is offline   Reply With Quote
Old 21st August 2018, 18:24   #26  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,531
mp3dom, out of interest if my approach would fit your scenario:
Does the bitrate behave properly if you add grain as I suggested in post 8 ?
(Since the source is not published, otherwise i would try myself.
My encodes were all Bluray 2-pass Level 4.1, slices = 4)
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..."
Emulgator is offline   Reply With Quote
Old 21st August 2018, 21:10   #27  |  Link
mp3dom
Registered User
 
Join Date: Jul 2003
Location: Italy
Posts: 1,135
I haven't tried an encode with your settings, but checked the effects of the added grain.
I think there's too much grain compared to the original, tbh. It may probably work but mainly because if the grain is always present throughout the stream, the problem is not "triggered" at all (i.e. x264 always keep the quantizers at a certain level).
I see this more like a way to "work around" the problem rather than to fix it.
It may work now, but I can't add grain to every stream...
mp3dom is offline   Reply With Quote
Old 22nd August 2018, 14:37   #28  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,531
That was what I wanted to know, no matter if it is your grain or my grain (the latter being barely noticeable at 2,2,1 delta pixels), it is only my workaround to exactly make "x264 always keep the quantizers at a certain level" and it does not fix the underlying problem, I agree right away from the start.
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..."
Emulgator is offline   Reply With Quote
Old 22nd August 2018, 15:50   #29  |  Link
mp3dom
Registered User
 
Join Date: Jul 2003
Location: Italy
Posts: 1,135
Yep, but in fact I thank you for the suggestion and effort, and I'll keep it as an alternative way just in case, but currently, as I work on commercial titles, I can't "deviate" from the real source too much, especially if the title is already available in other countries and can be easily compared.
Also, I guess the grain can't be too low or too much "invisible", because you can get the opposite effect (not enough quantization applied by x264 with blocky grain)
mp3dom is offline   Reply With Quote
Old 22nd August 2018, 22:14   #30  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Quote:
Originally Posted by mp3dom View Post
Yep, but in fact I thank you for the suggestion and effort, and I'll keep it as an alternative way just in case, but currently, as I work on commercial titles, I can't "deviate" from the real source too much, especially if the title is already available in other countries and can be easily compared.
Also, I guess the grain can't be too low or too much "invisible", because you can get the opposite effect (not enough quantization applied by x264 with blocky grain)


Did you try —level 4.0 —bitrate 25000 —slices 1? If that fixes the problem, that’d narrow down the issue to rate control of slices in particular.

I kind of hate slices; even though they work well most of the time, if you have significant variability in content characteristics between the slices, the encoder might wind up with some obvious QP differences along very visible horizontal lines.

IIRC Blu-ray only has slices for Level 4.1 to make it easier to make parallelized software decoders for PC playback. Frame-threaded decode is generally preferable, but it’s harder to do with only two B-frames.


Sent from my iPad using Tapatalk
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 9th September 2018, 15:29   #31  |  Link
mp3dom
Registered User
 
Join Date: Jul 2003
Location: Italy
Posts: 1,135
It happens even with level 4.0 and bitrate 25Mbps with 1 slice... but I think to have a little understood the culprit.
The problem appears only with VBV for a basic reason: given the maximum bitrate to be 40 Mbps and focusing on the last second only (where the scene change and the problem appear), the scene-change is at about 3/4 of that second. Previous scene (18 frames length) is bitrate hungry (crf-7 eats about 100 Mbps without VBV) while the remaining 6 frames (where the problem appears) is less demanding (crf-7 eats 40 Mbps without VBV), so x264 allocate the most of the bitrate to the hungry scene leaving not enough bitrate to encode the last 6 frames efficiently. The QPs on the grainy scene are near 15, while on the other scene are near 12. After that second, the scene "regain" its bitrate because the 40 Mbps are now spread across the same kind of scene, and not a new (totally different) one and the average qp is lowered to about 6. I guess adding grain to those 6 frames makes some difference mainly because I just "force" x264 to lower the quality on the grainy scene and give some more bitrate to the gradient scene.
mp3dom is offline   Reply With Quote
Old 11th September 2018, 05:19   #32  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Quote:
Originally Posted by mp3dom View Post
It happens even with level 4.0 and bitrate 25Mbps with 1 slice... but I think to have a little understood the culprit.
The problem appears only with VBV for a basic reason: given the maximum bitrate to be 40 Mbps and focusing on the last second only (where the scene change and the problem appear), the scene-change is at about 3/4 of that second. Previous scene (18 frames length) is bitrate hungry (crf-7 eats about 100 Mbps without VBV) while the remaining 6 frames (where the problem appears) is less demanding (crf-7 eats 40 Mbps without VBV), so x264 allocate the most of the bitrate to the hungry scene leaving not enough bitrate to encode the last 6 frames efficiently. The QPs on the grainy scene are near 15, while on the other scene are near 12. After that second, the scene "regain" its bitrate because the 40 Mbps are now spread across the same kind of scene, and not a new (totally different) one and the average qp is lowered to about 6. I guess adding grain to those 6 frames makes some difference mainly because I just "force" x264 to lower the quality on the grainy scene and give some more bitrate to the gradient scene.
Sounds like a job for two-pass encoding!
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 16th January 2019, 15:41   #33  |  Link
mp3dom
Registered User
 
Join Date: Jul 2003
Location: Italy
Posts: 1,135
I'm resuming this topic just to know if some dev had the chance to take a look into the problem that, depending on the source, may happens more or less often but it's very difficult and time consuming to fix afterwards, even with the use of zones.
Thanks.
mp3dom is offline   Reply With Quote
Old 16th January 2019, 17:48   #34  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Quote:
Originally Posted by mp3dom View Post
I'm resuming this topic just to know if some dev had the chance to take a look into the problem that, depending on the source, may happens more or less often but it's very difficult and time consuming to fix afterwards, even with the use of zones.
Thanks.
Yeah, it is a thorny one. Did you use a different encoder in the end?


Also, I'm curious if you tried --slices 1 --vbv-maxrate 25000. It might not make sense in practice, but could pinpoint whether this is a rate control issue specific to slices (which are a relatively rarely used x264 feature).

Also, what are the QPs before and after you get the hit? It's possible raising --qpmin some might reduce the initial VBV peak, allowing for a shallower valley to follow.

Lastly, perhaps try a third pass to see if that changes anything?
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 16th January 2019, 19:34   #35  |  Link
mp3dom
Registered User
 
Join Date: Jul 2003
Location: Italy
Posts: 1,135
I was able to fix it a bit (the problem is still there, but less evident) adding a cetain amount of grain to the image, but this clearly is a workaround and not the right way to fix it, since the grain was not there in the source! Also, adding grain is not always enough, it depends on the scene itself and with totally flat sources, adding grain on selected scenes is a no-way... too much evident that there's something wrong.
The problem is triggered when using vbv, 1 slice or 4 slices doesn't make any difference, and the same apply to the use of a third pass or lowering the threads.
During and after the "hit" there's a big jump in QP values, I saw a difference of 14-15 on some cases and since the scene is exactly the same, it's clearly visible especially on gradients/shades, albeit the problem last for 1 second.

Last edited by mp3dom; 17th January 2019 at 11:01.
mp3dom is offline   Reply With Quote
Old 16th January 2019, 19:50   #36  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,308
As far as i know, even if trigging the issue is not easy, you have provided the devs specific materials to reproduce the issue since a little time now.
__________________
My github.

Last edited by jpsdr; 16th January 2019 at 21:37.
jpsdr is offline   Reply With Quote
Old 16th January 2019, 19:55   #37  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Quote:
Originally Posted by mp3dom View Post
I was able to fix it a bit (the problem is still there, but less evident) adding a cetain amount of grain to the image, but this clearly is a workaround and not the right way to fix it, since the grain was not there in the source! Also, adding grain is not always enough, it depends by the scene itself and with totally flat sources, adding grain on selected scenes is a no-way... too much evident that there's something wrong.
The problem is triggered when using vbv, 1 slice or 4 slices doesn't make any difference, and the same apply to the use of a third pass or lowering the threads.
During and after the "hit" there's a big jump in QP values, I saw a difference of 14-15 on some cases and since the scene is exactly the same, it's clearly visible especially on gradients/shades, albeit the problem last for 1 second.
Yeah, sounds like a fundamental rate control issue. Blu-ray uses a lot shorter GOPs than "normal" x264 scenarios, which limits the scope of --rc-lookahead, which could factor in. A real fix will require dev work.

A temp fix would probably entail reducing the initial peak, so the following valley doesn't need to go so low. Something like --qpmin 8 or --nr 500 might be worth trying. Hopefully lower values could work in practice.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 17th January 2019, 09:41   #38  |  Link
mp3dom
Registered User
 
Join Date: Jul 2003
Location: Italy
Posts: 1,135
Quote:
Originally Posted by jpsdr View Post
As far as i know, even if trigging the issue is not easy, you have provided the devs specific materials to reproduce the issue since a little time now.
3-4 months ago I think...

Quote:
Originally Posted by benwaggoner View Post
A temp fix would probably entail reducing the initial peak, so the following valley doesn't need to go so low. Something like --qpmin 8 or --nr 500 might be worth trying. Hopefully lower values could work in practice.
Yeah, but it's very time-consuming because if you have 5-6 of these problems in a single movie, it's hard to fix all of them and you're going to encode the movie a lot of time before getting acceptable results.

Last edited by mp3dom; 17th January 2019 at 09:44.
mp3dom is offline   Reply With Quote
Old 17th January 2019, 19:45   #39  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Quote:
Originally Posted by mp3dom View Post
3-4 months ago I think...


Yeah, but it's very time-consuming because if you have 5-6 of these problems in a single movie, it's hard to fix all of them and you're going to encode the movie a lot of time before getting acceptable results.
Yeah, I don't see any production-ready solution without an actual bug fix.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 20th January 2019, 22:55   #40  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,531
Out of curiosity, while getting back to adding grain as I suggested seemed to help,
has anyone tried x264 incorporated in TMPGEnc AuthoringWorks 5 regarding that matter?
I found that private x264 build quite nice for the few occasions I used it on
but did not bother doing elaborated comparisons regarding the bitrate drop problem back then...
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..."
Emulgator is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 12:19.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.