x265 HEVC Encoder

[LigH]

Why not running the 2nd pass of the 2-pass encode again?

[RainyDog]

Originally Posted by LigH

Why not running the 2nd pass of the 2-pass encode again?

Thanks LigH. Well, that's kind of what I'm asking... Will a 1-pass ABR encode using the existing stats file produce the same result as if it were a 2nd pass?

Basically I'm using staxrip as the GUI and the encode options are 2-pass, Bitrate (1-pass ABR) and Quality (CRF). If I select 2-pass again then it will just start again and run another 1st pass creating a new stats file.

But if I select Bitrate ABR mode and add --pass 2 and --stats as custom encode options, when I set it going it takes a few seconds to read the stats file and then states --stats read once it starts. So its definitely using the stats file but my concern is is it using the 'proper' 2nd pass algorithm and not the poorer 1-pass ABR algorithm which I believe is internally different to the other modes...

Thanks.

[LigH]

I don't know which GUI you are using (x265 is a CLI encoder). If you can't select "2-pass, 1st pass" and "2-pass, 2nd pass" separately, but only "2-pass" as a bundle, then it may not be very helpful for your case.

The workaround you describe may work, but I didn't think much about it. After you tell us which GUI you are using, we might know a better solution, with or without this GUI.

By the way, the 1st pass may have been so quick, it could have been ran in the meantime...

[RainyDog]

Originally Posted by LigH

I don't know which GUI you are using (x265 is a CLI encoder). If you can't select "2-pass, 1st pass" and "2-pass, 2nd pass" separately, but only "2-pass" as a bundle, then it may not be very helpful for your case.

The workaround you describe may work, but I didn't think much about it. After you tell us which GUI you are using, we might know a better solution, with or without this GUI.

By the way, the 1st pass may have been so quick, it could have been ran in the meantime...

I'm using Staxrip.

Well, the reason I didn't necessarily want to re-run the 1st pass is that I use custom 1st pass settings which are almost the same as the 2nd pass settings with a few changes to speed it up a bit. So it this instance the 1st pass took aalmost 6 hours... so if I can re-use it's stats file then it would save me 6 hours at least anyway!

Would also like to know for future reference really, in case it happens again.

Thanks.

[nevcairiel]

So just run the 2nd pass again using the stats file? Thats part of what makes the separate passes so nice, you can just run the 2nd pass again. Ordinarily one might use it to tune settings or whatnot, but if it failed you can also just start over.

[RainyDog]

Originally Posted by nevcairiel

So just run the 2nd pass again using the stats file? Thats part of what makes the separate passes so nice, you can just run the 2nd pass again. Ordinarily one might use it to tune settings or whatnot, but if it failed you can also just start over.

Hi nevcairiel, yes that's what I think I'm doing.

But I was just unsure whether I'd be unwittingly running an encode using ABR rate control mode as that's the option I'm having to select in Staxrip even though it's using a stats file already created from a successful 1st pass.

I believe ABR rate control mode is not recommended under any circumstances unless you must hit a certain bitrate/size and simply dont have the time for a 1st pass as it uses a different interal algorithm to CRF or 2pass encodes...

[nevcairiel]

You should probably ask the authors of that GUI how to run only a 2nd pass again. With the x265 CLI interface its pretty straight forward.

[RainyDog]

Originally Posted by nevcairiel

You should probably ask the authors of that GUI how to run only a 2nd pass again. With the x265 CLI interface its pretty straight forward.

Do you just add --pass 2 and --stats to the commnd line using the CLI interface?

[nevcairiel]

Basically yes, together with all other encoding options.

[RainyDog]

Originally Posted by nevcairiel

Basically yes, together with all other encoding options.

Thanks. Yeah, that's what I've done in the Staxrip custom command line options section.

I set it going before I headed off to work this morning and it definitely read/loaded the stats file before starting the encode and stated "stats-read" on the GUI output display anyway. Which it always does on a 2nd pass and states "stats-write" on a 1st pass.

[alex1399]

Hey guys, you might overlook the three pass encode for a fine result. The 1-pass ABR encode have two meaning, the --pass 1 activated first pass ABR encode and the initial ABR encode without --pass 1 activated. Once the --pass 1 is not included in the first pass ABR encode, add the --pass 1 and the --pass 2 for the following second and third pass encode. Else you could add --pass 3 and --pass 2 for the following three pass encode.

[benwaggoner]

Originally Posted by alex1399

Hey guys, you might overlook the three pass encode for a fine result. The 1-pass ABR encode have two meaning, the --pass 1 activated first pass ABR encode and the initial ABR encode without --pass 1 activated. Once the --pass 1 is not included in the first pass ABR encode, add the --pass 1 and the --pass 2 for the following second and third pass encode. Else you could add --pass 3 and --pass 2 for the following three pass encode.

Has anyone seen meaningful improvements from a third pass in x265? Since the 1st pass is a slow firstpass by default, I'd guess the gains would be less than in x264.

[WhatZit]

Originally Posted by benwaggoner

Has anyone seen meaningful improvements from a third pass in x265?

I toyed with this for several months, and was hard-pressed to identify any tangible benefit, even using naughty flick-screen frame grabs as comparison.

There was only ever the most microscopic detail improvements in low-contrast/high motion over a 2-pass, but these were easily homogenised out by the video processors in my TV's.

[benwaggoner]

Originally Posted by WhatZit

I toyed with this for several months, and was hard-pressed to identify any tangible benefit, even using naughty flick-screen frame grabs as comparison.

There was only ever the most microscopic detail improvements in low-contrast/high motion over a 2-pass, but these were easily homogenised out by the video processors in my TV's.

I’ve only seen tangible improvements in 2-pass at low —bitrate —vbv-maxrate and -vbv-maxsize , OR when —vbv-maxrate is significantly higher than —bitrate

[RainyDog]

On the subject of multi-pass encodes, has anyone done much testing of the multi-pass-opt options --multi-pass-opt-analysis and --multi-pass-opt-distortion?

--multi-pass-opt-analysis, --no-multi-pass-opt-analysis

Enable/Disable multipass analysis refinement along with multipass ratecontrol. Based on the information stored in pass 1, in subsequent passes analysis data is refined and also redundant steps are skipped. In pass 1 analysis information like motion vector, depth, reference and prediction modes of the final best CTU partition is stored for each CTU. Multipass analysis refinement cannot be enabled when ‘analysis-save/analysis-load’ option is enabled and both will be disabled when enabled together. This feature requires ‘pmode/pme’ to be disabled and hence pmode/pme will be disabled when enabled at the same time.

--multi-pass-opt-distortion, --no-multi-pass-opt-distortion

Enable/Disable multipass refinement of qp based on distortion data along with multipass ratecontrol. In pass 1 distortion of best CTU partition is stored. CTUs with high distortion get lower(negative)qp offsets and vice-versa for low distortion CTUs in pass 2. This helps to improve the subjective quality. Multipass refinement of qp cannot be enabled when ‘analysis-save/analysis-load’ option is enabled and both will be disabled when enabled together. ‘multi-pass-opt-distortion’ requires ‘pmode/pme’ to be disabled and hence pmode/pme will be disabled when enabled along with it.

Seems that opt-analysis is more for speeding up subsequent passes by storing more data during the 1st pass for re-use. So... I think this might be more safely suited to multi-pass encodes where all passes use the exact (or almost) same settings?

And opt-distortion refines and re-balances QP distribution during subsequent passes based on info gathered in the 1st pass. Which sounds like a no brainer to use as long as it performs as it should... And perhaps even provides a genuine quality advantage to multi-pass encodes over CRF.

[alex1399]

The three pass encode is a refine of two pass encode where the original second pass encode is put at the end and an additional pass of slice-type / bit distribution encode inserted after the first pass. It is often to see that x265 places b-frame on some arbitrary location (not sub-optimal at all) even when the Quantization Parameter is not fluctuating everywhere. For the first pass encode, it get worse when x265 has to guess the target Rate Factor and the slice-type decision would be interfered by those Quantization Parameter fluctuation. The second pass in the two pass encode simply read the slice-type distribution from the first pass without any enhancement. The b-frame distribution is much consistent in the three pass encode. The three pass encode would place those b-frame like a specific case of two pass encode that utilize Constant Rate Factor encode for the first pass and Average Bit Rate encode for the second pass. That's why I prefer three pass encode.

For the --multi-pass-opt-analysis and the --multi-pass-opt-distortion, x265 prefers the Psycho-visual Rate Distortion measure to bias the Quantization Parameter in advance. I'm not a fan of those stuff. The compression efficiency based on non Psycho-visual metrics, for example the PSNR per bit, should still be a fundamental parts of encoder development. I'm aware that some encoder could trick on PSNR, but it doesn't sway my decision making.

[benwaggoner]

Originally Posted by RainyDog

On the subject of multi-pass encodes, has anyone done much testing of the multi-pass-opt options --multi-pass-opt-analysis and --multi-pass-opt-distortion?





Seems that opt-analysis is more for speeding up subsequent passes by storing more data during the 1st pass for re-use. So... I think this might be more safely suited to multi-pass encodes where all passes use the exact (or almost) same settings?

And opt-distortion refines and re-balances QP distribution during subsequent passes based on info gathered in the 1st pass. Which sounds like a no brainer to use as long as it performs as it should... And perhaps even provides a genuine quality advantage to multi-pass encodes over CRF.

Yes, that's the one that should improve quality, and I imagine it should always be used in any second pass, particularly when doing CBR or having a constrained VBV. In the cases where I found 2-pass helpful, I was using --opt-distortion.

[RainyDog]

Originally Posted by Magik Mark

Ma & LigH,

Found the error in 2pass encoding:

Multipass analysis refinement along with multipass rate control
Multipass refinement of qp based on distortion data

If these two are deactivated everything is ok

Maybe the syntax has changed?

Found yesterday that this bug is still present in x265 2.8+66.

1st pass runs ok and creates the usual stats file plus a huge (11.5gb!) analysis file. Then crashes when it tries to run the 2nd pass if --multi-pass-opt-distortion has been set.

Are the developers aware of this? Thanks.

[RainyDog]

Here's my Staxrip log for the error when trying to start the 2nd pass when --multi-pass-opt-distortion is set.

"C:\Portable Programs\StaxRip\Apps\avs2pipemod\avs2pipemod64.exe" -y4mp "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976).avs" | "C:\Portable Programs\StaxRip\Apps\x265\x265.exe" --pass 1 --preset slow --profile main10 --output-depth 10 --rd 2 --psy-rd 2.0 --psy-rdoq 1.0 --subme 2 --aq-mode 3 --aq-strength 0.8 --qcomp 0.7 --bframes 6 --ipratio 1.2 --pbratio 1.1 --ctu 32 --no-cutree --merange 32 --qg-size 8 --no-rect --rc-lookahead 40 --lookahead-slices 0 --no-open-gop --early-skip --fast-intra --me dia --cbqpoffs -2 --crqpoffs -2 --deblock -2:-2 --no-sao --no-strong-intra-smoothing --no-constrained-intra --rdpenalty 1 --limit-tu 3 --tu-intra-depth 3 --tu-inter-depth 3 --multi-pass-opt-distortion --analysis-reuse-file "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976)_out.analysis" --bitrate 9000 --no-rect --no-cutree --weightb --no-open-gop --no-sao --no-strong-intra-smoothing --no-constrained-intra --frames 128857 --y4m --stats "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976).stats" --output NUL -

avs2pipemod[info]: writing 128857 frames of 24000/1001 fps, 1920x1040,
sar 0:0, YUV-420-planar-8bit progressive video.
y4m [info]: 1920x1040 fps 24000/1001 i420p8 unknown frame count
raw [info]: output file: NUL
x265 [info]: HEVC encoder version 2.8+66-88ee12651e30
x265 [info]: build info [Windows][MSVC 1915][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main 10 profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 3 / wpp(33 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: ME / range / subpel / merge : dia / 32 / 2 / 3
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 40 / 5.00
x265 [info]: Cb/Cr QP Offset : -2 / -2
x265 [info]: Intra 32x32 TU penalty type : 1
x265 [info]: Lookahead / bframes / badapt : 40 / 6 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 4 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 3 / 0.8 / 8 / 0
x265 [info]: Rate Control / qCompress : ABR-9000 kbps / 0.70
x265 [info]: tools: limit-modes rd=2 psy-rd=2.00 rdoq=2 psy-rdoq=1.00
x265 [info]: tools: early-skip rskip limit-tu=3 signhide tmvp fast-intra
x265 [info]: tools: deblock(tC=-2:B=-2) stats-write
avs2pipemod[info]: finished, wrote 128857 frames [100%].
avs2pipemod[info]: total elapsed time is 20045.940 sec.
x265 [info]: frame I: 1114, Avg QP:25.00 kb/s: 16601.32
x265 [info]: frame P: 23173, Avg QP:25.86 kb/s: 10044.17
x265 [info]: frame B: 104570, Avg QP:26.85 kb/s: 8597.98
x265 [info]: Weighted P-Frames: Y:8.1% UV:7.0%
x265 [info]: Weighted B-Frames: Y:8.3% UV:6.6%
x265 [info]: consecutive B-frames: 7.7% 1.6% 4.1% 6.5% 13.2% 53.3% 13.7%
encoded 128857 frames in 20047.53s (6.43 fps), 8927.25 kb/s, Avg QP:26.65

Start: 21:13:51
End: 02:47:59
Duration: 05:34:07

---------- Error Video encoding second pass using x265 2.6+31 ----------

Video encoding second pass using x265 2.6+31 failed with exit code: -1073741819 (0xC0000005)

The exit code might be a system error code: The instruction at 0xp referenced memory at 0xp. The memory could not be s.


------------- Video encoding second pass using x265 2.6+31 -------------

"C:\Portable Programs\StaxRip\Apps\avs2pipemod\avs2pipemod64.exe" -y4mp "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976).avs" | "C:\Portable Programs\StaxRip\Apps\x265\x265.exe" --pass 2 --preset slow --profile main10 --output-depth 10 --rd 4 --psy-rd 2.0 --psy-rdoq 1.0 --subme 4 --aq-mode 3 --aq-strength 0.8 --qcomp 0.7 --bframes 6 --ipratio 1.2 --pbratio 1.1 --ctu 32 --weightb --no-cutree --merange 32 --qg-size 8 --no-rect --rc-lookahead 40 --lookahead-slices 0 --no-open-gop --cbqpoffs -2 --crqpoffs -2 --deblock -2:-2 --no-sao --no-strong-intra-smoothing --no-constrained-intra --rdpenalty 1 --limit-tu 3 --tu-intra-depth 3 --tu-inter-depth 3 --multi-pass-opt-distortion --analysis-reuse-file "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976)_out.analysis" --bitrate 9000 --no-rect --no-cutree --no-open-gop --no-sao --no-strong-intra-smoothing --no-constrained-intra --frames 128857 --y4m --stats "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976).stats" --output "C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976)_out.hevc" -

avs2pipemod[info]: writing 128857 frames of 24000/1001 fps, 1920x1040,
sar 0:0, YUV-420-planar-8bit progressive video.
y4m [info]: 1920x1040 fps 24000/1001 i420p8 unknown frame count
raw [info]: output file: C:\Encoding\Source\God Told Me To (1976)_temp\God Told Me To (1976)_out.hevc
x265 [info]: HEVC encoder version 2.8+66-88ee12651e30
x265 [info]: build info [Windows][MSVC 1915][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main 10 profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 3 / wpp(33 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: ME / range / subpel / merge : star / 32 / 4 / 3
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 40 / 5.00
x265 [info]: Cb/Cr QP Offset : -2 / -2
x265 [info]: Intra 32x32 TU penalty type : 1
x265 [info]: Lookahead / bframes / badapt : 40 / 6 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 4 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 3 / 0.8 / 8 / 0
x265 [info]: Rate Control / qCompress : ABR-9000 kbps / 0.70
x265 [info]: tools: limit-modes rd=4 psy-rd=2.00 rdoq=2 psy-rdoq=1.00 rskip
x265 [info]: tools: limit-tu=3 signhide tmvp deblock(tC=-2:B=-2) stats-read
avs2pipemod[info]: finished, wrote 55 frames [0%].
avs2pipemod[info]: total elapsed time is 7.488 sec.
avs2pipemod[error]: only wrote 55 of 128857 frames.

Ignore the parts about using x265 2.6+31, this is just the version of x265 that was bundled with the last official release of Staxrip. I'm using 2.8+66.

[Ma]

Thanks, now I'm able to reproduce the problem.
My first observation is:
8bit+10bit+12bit versions of x265 hangs/not working in second pass
10bit or 10bit+8bit versions of x265 works OK

Under investigation...