[Dark Shikari]

Originally Posted by DeeGee

If I'm understanding correctly a new commit in x264 sources, your problem might be already fixed. You just need to find a new enough x264 build to test if it helps.
https://github.com/DarkShikari/x264-...bab6c2b4610454

This commit most likely has nothing to do with the problem.

I did try setting a max quantizer (--qpmax), but that seems unpredictable; I started getting buffer underflows.

Setting QP-max will cause buffer underflows because it prevents VBV from doing its job.

If you have a small sample of the video that triggers the problem, please post it and a set of encoding settings that consistently produce the problem (along with pointing me towards whatever frame numbers have the issue).

If you really need a constant bitrate video at the expense of quality, try using CBR mode.

Let me clear up some misconceptions here, because the previous post was extremely misleading and largely incorrect:

Originally Posted by LoRd_MuldeR

Though it has to be noted that VBV allows (short) bitrate spikes above VBV Maxrate...

Only if you incorrectly define VBV maxrate.

Originally Posted by LoRd_MuldeR

BTW: I don't think VBV is the problem here. If VBV effects quality, you would rather see this in "high complexity" (high bitrate) scenes.

This is not necessarily true.

Originally Posted by LoRd_MuldeR

The first pass is basically 1-Pass ABR encode, so no big surprise that the resulting bitrate is relatively constant.

This is not necessarily true. 1-pass bit distribution may be suboptimal, but it is not "relatively constant".

Originally Posted by LoRd_MuldeR

In the second pass, 2-Pass rate-control is applied to distribute the available bits over the complete video. And of course "low complexity" scenes will get fewer bits than "high complexity" scenes. If you think too few bits are assigned to the "low complexity" scenes, you'll have to tweak x264's 2-Pass RC algorithm.

The part of x264's RC algorithm that affects bit distribution, macroblock-tree, affects both passes identically. It has absolutely nothing to do with 1-pass or 2-pass.

Originally Posted by LoRd_MuldeR

I think "--qcomp" might be one parameter you could try to tweak here...

This affects both passes identically.