x265 HEVC Encoder

[Ma]

Bisection is one of the most common method of finding bugs.
In source file common/x86/asm-primitives.cpp it is function pointers assignment with avx512 code -- from line # 4696 to 5385.
We can turn off (comment) first half of this functions (x265-1.exe) and then second half (x265-2.exe).
You can download avx512-bisect1.7z VS 2015 binaries with diff files -- if x265.exe (from clean sources) hangs, please try x265-1.exe and then x265-2.exe -- it should be that one of them hangs and second not.
Please report back the results (which from the 3 binaries hangs and which not).

[Xizer]

Originally Posted by Ma

Bisection is one of the most common method of finding bugs.
In source file common/x86/asm-primitives.cpp it is function pointers assignment with avx512 code -- from line # 4696 to 5385.
We can turn off (comment) first half of this functions (x265-1.exe) and then second half (x265-2.exe).
You can download avx512-bisect1.7z VS 2015 binaries with diff files -- if x265.exe (from clean sources) hangs, please try x265-1.exe and then x265-2.exe -- it should be that one of them hangs and second not.
Please report back the results (which from the 3 binaries hangs and which not).

All three binaries crash with the avx512 flag I'm afraid.

Is there additional information that can be provided for debug?

[Ma]

Originally Posted by Xizer

All three binaries crash with the avx512 flag I'm afraid.

Is there additional information that can be provided for debug?

Thanks for the info.

Could you post results of the command
x265 -V && x265 --asm avx2 -V && x265 --asm avx512 -V
For example in my system it looks like

F:\x265p\ma\avx512>x265 -V && x265 --asm avx2 -V && x265 --asm avx512 -V
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512

This bug could be not related directly to avx512 code -- could you check if it hangs if you use '--asm avx2' instead of '--asm avx512' (it is important to use --asm avx2 option).
------------------------------------------------------
Your OS is Win 8.1 that do not support avx512. This bug in x265 is not technical but conceptual -- avx512 is not auto recognized by default so option '--asm avx512' should not turn on avx512 without any check.
You can test file avx512-patch.7z with x265 that check what is supported by CPU & OS up to avx512 if option '--asm avx512' is used.
On Win 8.1 it should work with '--asm avx512' exactly like without this option, on Win 10 it should turn on avx512 if you have CPU with avx512 and you use option '--asm avx512'.

[Bhavnahari]

Originally Posted by jlpsvk

any suggested quality wise settings recommendation for 4K HDR encoding? with CRF ie 17?

If the data that is being reused comes from encoding a downscaled video (scale-factor=2), it does not make a lot of sense to use intra/inter refinement levels 0/1 as levels 0 and 1 reuse most of the information from the previous pass with no or minimal re-evaluation of analysis information. This can have a huge impact on the quality of the encode.

We have observed that --refine-inter=3 and --refine-intra=4 gives the best quality, even better than standalone x265 encodes in some cases, with a performance gain of up to 1.8X.

For 4K HDR content, you will have to modify the display settings based on the monitor. Please refer the docs for more information - http://x265.readthedocs.io/en/defaul...mation-options

[jlpsvk]

Originally Posted by Bhavnahari

If the data that is being reused comes from encoding a downscaled video (scale-factor=2),
it does not make a lot of sense to use intra/inter refinement levels 0/1 as levels 0 and 1 reuse most of the information from
the previous pass with no or minimal re-evaluation of analysis information. This can have a huge impact on the quality of the encode.

We have observed that --refine-inter=3 and --refine-intra=4 gives the best quality, even better than standalone x265 encodes in some cases,
with a performance gain of up to 1.8X.

My settings then...

--crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock
--no-sao --no-strong-intra-smoothing --high-tier --refine-inter 3 --refine-intra 4

Of course, display settings are entered too...

Warning:

x265 [warning]: Intra refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling intra refine.
x265 [warning]: Inter refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling inter refine.

Can it be used with CRF? Or? Some parameters missing?

[Loomes]

So far no success for me running the avx512 option on my 7820x on Windows 10 64it. I tried

ffmpeg -i source.mkv -f yuv4mpegpipe - | x265.exe --preset slow --asm avx512 --crf 21 --y4m - -o dest.h265

which crashes immediately. Using --asm avx2 works fine.

I ran

x265.exe -V && x265.exe --asm avx2 -V && x265.exe --asm avx512 -V

and it showed

x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512

Originally Posted by Ma

You can test file avx512-patch.7z with x265 that check what is supported by CPU & OS up to avx512 if option '--asm avx512' is used.

I tried from that package x265.exe, x265-1.exe and x265-2.exe. All of them crashed immediately. I am using the latest nightly version of ffmpeg.
Am I missing something?

[Asmodian]

Originally Posted by Loomes

I tried from that package x265.exe, x265-1.exe and x265-2.exe. All of them crashed immediately. I am using the latest nightly version of ffmpeg.
Am I missing something?

I had crashes using the builds foxyshadis posted but I was successful using LigH's build from this post. Win10 and an i9-7900X. Have you tried them?

[Loomes]

Originally Posted by Asmodian

I had crashes using the builds foxyshadis posted but I was successful using LigH's build from this post. Win10 and an i9-7900X. Have you tried them?

I did now but it also results in an instant crash. Are you using a ffmpeg pipe like me or what is your command line?

[Ma]

Originally Posted by Loomes

I tried from that package x265.exe, x265-1.exe and x265-2.exe. All of them crashed immediately. I am using the latest nightly version of ffmpeg.
Am I missing something?

x265-1.exe == x265-2.exe == x265.exe from package avx512-bisect1.7z (I commented out code for 8-bit encoding and compiled 10-bit version, sorry for that). So this package is to forget.

I want to precise: x265.exe from package avx512-patch.7z gives in your OS output:
x265 --asm avx512 -V
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
?

[Loomes]

Originally Posted by Ma

I want to precise: x265.exe from package avx512-patch.7z gives in your OS output:
x265 --asm avx512 -V
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
?

Confirmed. I tried again on Windows 10 64bit with an 7820x using the command line shown above.

[Asmodian]

Originally Posted by Loomes

I did now but it also results in an instant crash. Are you using a ffmpeg pipe like me or what is your command line?

No, actually I was using avs2pipemod.exe (ver 1.1.1 Aug 15, 2016) piping to x265.exe with y4m:
C:\Tools\avs2pipemod.exe -y4mp=1:1 %1 | C:\Tools\x265.exe --input - --y4m -o "D:\Temp\%~n1.mkv" --asm avx512 --preset veryslow --crf 18.5 --output-depth 10

edit:
"x265 --asm avx512 -V" gives:
x265 [info]: HEVC encoder version 2.7+340-aa9102400f24
x265 [info]: build info [Windows][GCC 7.3.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512

[Ma]

Originally Posted by Loomes

Confirmed. I tried again on Windows 10 64bit with an 7820x using the command line shown above.

Thanks for confirmation -- it looks like your CPU & OS supporting avx512.

I've prepared some binaries to narrow this bug -- avx512-bisect2.7z.
x265-0.exe -- 10-bit encoder with commented out all avx512 function pointers (it works on my i7 8700 with --asm avx512 option). It is for check if the bug is before avx512 code is running but the execution path is exactly like in avx512 case.

x265-01.exe -- assigned only first 1/4 function pointers with avx512 code.
x265-02.exe -- assigned only second 1/4 function pointers with avx512 code.
x265-03.exe -- assigned only third 1/4 function pointers with avx512 code.
x265-04.exe -- assigned only last 1/4 function pointers with avx512 code.

Could you test these 5 exe's and report back which works and which not?

[Loomes]

Originally Posted by Ma

Could you test these 5 exe's and report back which works and which not?

I'm home in about 6 hours and will do. By the way: Is it correct that --asm avx512 will not work on Windows 7/8 64bit, no matter what?

[nevcairiel]

Originally Posted by Loomes

Is it correct that --asm avx512 will not work on Windows 7/8 64bit, no matter what?

That would appear to be the case. If you force avx512 usage through --asm avx512 on an other OS it'll just crash.

Thats really the problem with using the --asm option, it overrides the CPU/OS feature detection, a different option to enable it when present would be better.

[Loomes]

Originally Posted by Ma

Could you test these 5 exe's and report back which works and which not?

I am afraid, none of the x265-xx.exe worked in my Windows 10 and crashed instantly. But I found something else: I tried to use directly the libx265 of ffmpeg by

ffmpeg-2018-04-18.exe -i source.mkv -c:v libx265 -x265-params crf=21 -preset slow -pix_fmt yuv420p10le -bsf hevc_mp4toannexb -f rawvideo target.h265

and it also crashed instantly. Then I changed this new ffmpeg-2018-04-18.exe for an older one ffmpeg-2018-03-24.exe and it worked. Then I ran again your x265-xx.exe files with the older ffmpeg-2018-03-24.exe but they all crashed again with

ffmpeg-2018-03-24.exe -i source.mkv -f yuv4mpegpipe - | x265-0.exe --preset slow --crf 21 --y4m - -o target.h265

Even more strange that ffmpeg-2018-04-18.exe works flawlessly on Windows 7 and so the x265-0.exe of your files does (it even claims to make use of AVX512 when it starts). All the other x265-0x.exe crashed immediately on Windows 7.

This is really weird. Let me know if I can provide you any more info of my system or do some more tests.

[Ma]

Thanks for the details. It is quite a riddle.
On Win7 everything is OK, x265-0 should work (all avx512 code is not active), x265-0x should crash.
On Win10 it is interesting -- in your 2 examples it is no '--asm avx512' option. It crashes without avx512 too?
Is something special with your Win10, for example it works in VirtualBox?

[Loomes]

Originally Posted by Ma

On Win10 it is interesting -- in your 2 examples it is no '--asm avx512' option. It crashes without avx512 too?

Yes, I had --asm avx512 included in the first run and all crashed. Then I had the idea that it's about ffmpeg and let it run without the asm option but well, did not work either.

Originally Posted by Ma

Is something special with your Win10, for example it works in VirtualBox?

No, not at all. It is Windows 10 Professional 64bit, nothing special about it as far as I know. The system drive has 14GB free space, the drive where x265, ffmpeg etc. is running with 80G free space. As I said, it works fine with the older ffmpeg version or older x265.exe.
Is it possible that there is something special about the 7820x CPU?

[Ma]

Originally Posted by Loomes

Is it possible that there is something special about the 7820x CPU?

It is possible that there are hardware problems -- easiest way to check this out is Prime95 https://www.mersenne.org/download/
If you run this app with button 'Just Stress Testing' it should find memory/CPU problems (in file 'results.txt' should be only [date] and XXX passed! entries).

There was some changes in ffmpeg, so you can test ffmpeg release 3.4.2 -- it should be different approach to thread synchronization.

There was some reports about x265 hangs with pipe input from ffmpeg but it is hard to reproduce -- maybe there is bug in x265 that is not fixed yet.

[Bhavnahari]

Originally Posted by jlpsvk

My settings then...

--crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock
--no-sao --no-strong-intra-smoothing --high-tier --refine-inter 3 --refine-intra 4

Of course, display settings are entered too...

Warning:

x265 [warning]: Intra refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling intra refine.
x265 [warning]: Inter refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling inter refine.

Can it be used with CRF? Or? Some parameters missing?

refine-inter and refine-intra features may be used with any rate control techniques including ABR, CRF and CQP. However, as indicated by the warnings, these features are enabled only for scaled save/load encodes. The "save" run will encode the content of resolution WxH (1920x1080, say) and generate an analysis.dat file. This analysis information is scaled up by the factor and used by the "load" run which works on the content of resolution 2Wx2H (3840x2160, say). Using the analysis information from a low-res encode to encode high-res content will impact the quality of the encode hence we try to minimize the repercussions by incorporating various intra and inter refinement strategies.

Your CLI should be:
HD encode : --crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-save <filename> --analysis-reuse-level 10 --scale-factor 2

UHD encode: -crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-load <filename> --analysis-reuse-level 10 --scale-factor 2 --refine-inter 3 --refine-intra 4

[jlpsvk]

Originally Posted by Bhavnahari

refine-inter and refine-intra features may be used with any rate control techniques including ABR, CRF and CQP. However, as indicated by the warnings, these features are enabled only for scaled save/load encodes. The "save" run will encode the content of resolution WxH (1920x1080, say) and generate an analysis.dat file. This analysis information is scaled up by the factor and used by the "load" run which works on the content of resolution 2Wx2H (3840x2160, say). Using the analysis information from a low-res encode to encode high-res content will impact the quality of the encode hence we try to minimize the repercussions by incorporating various intra and inter refinement strategies.

Your CLI should be:
HD encode : --crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-save <filename> --analysis-reuse-level 10 --scale-factor 2

UHD encode: -crf 17 --profile main10 --level-idc 5.1 --output-depth 10 --ctu 32 --amp --vbv-bufsize 160000 --vbv-maxrate 160000 --me star
--max-merge 5 --rc-lookahead 40 --lookahead-slices 4 --gop-lookahead 34 --ref 5 --hdr --hdr-opt --repeat-headers --no-info --no-deblock --no-sao --no-strong-intra-smoothing --high-tier --analysis-load <filename> --analysis-reuse-level 10 --scale-factor 2 --refine-inter 3 --refine-intra 4

Thanks!!! So it won't downscale and upscale video? Only stats? Right? So it's better for retain quality/details or not?