Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 22nd May 2018, 07:05   #6141  |  Link
user1085
Registered User
 
Join Date: Apr 2018
Posts: 9
Anyone know how to build x265 2.8 with AVX512 enabled on Ubuntu?

I built x265 from 2.8 release tarball but x265 -V doesn't show me it's using AVX512 capability on a Intel Xeon

Code:
x265 [info]: HEVC encoder version 2.8
x265 [info]: build info [Linux][GCC 7.3.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
Here's the CPU info

Code:
uso@xvmub:~/x265_2.8/build/linux$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              2
On-line CPU(s) list: 0,1
Thread(s) per core:  2
Core(s) per socket:  1
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
Stepping:            4
CPU MHz:             2693.677
BogoMIPS:            5387.35
Virtualization:      VT-x
Hypervisor vendor:   Microsoft
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            33792K
NUMA node0 CPU(s):   0,1
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti tpr_shadow vnmi ept vpid fsgsbase bmi1 hle avx2 smep bmi2 erms rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves
user1085 is offline   Reply With Quote
Old 22nd May 2018, 07:22   #6142  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 5,309
AVX-512 is only enabled if you explicitly add --cpu avx512 to the command line; without it is not even checked for in CPU capabilities, apparently... so try:

Code:
x265 --cpu avx512 -V
Is it reported then?
__

Quote:
Originally Posted by user1085 View Post
Do you also have a Linux build?
No, I was a) quite confident that Linux users know how to install and use GNU compilers (it's good enough documented on Multicoreware's Bitbucket), and b) not sure how portable one build under one specific distro is. Windows builds are quite portable, and knowledge about compiling from sources is probably not so common among Windows users.
__________________

German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 22nd May 2018, 07:31   #6143  |  Link
user1085
Registered User
 
Join Date: Apr 2018
Posts: 9
Quote:
Originally Posted by LigH View Post
AVX-512 is only enabled if you explicitly add --cpu avx512 to the command line; without it is not even checked for in CPU capabilities, apparently... so try:

Code:
x265 --cpu avx512 -V
Is it reported then?
__



No, I was a) quite confident that Linux users know how to install and use GNU compilers (it's good enough documented on Multicoreware's Bitbucket), and b) not sure how portable one build under one specific distro is. Windows builds are quite portable, and knowledge about compiling from sources is probably not so common among Windows users.
No worries I figured out how to compile, didn't see the instructions earlier

Re:avx512, this worked for me
Code:
./x265 --asm avx512 -V
user1085 is offline   Reply With Quote
Old 22nd May 2018, 07:34   #6144  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 5,309
Oops, should have read the manual, "asm" instead of "cpu".

Thank you for confirming anyway.
__________________

German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 22nd May 2018, 20:53   #6145  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 2,431
Quote:
Originally Posted by Sagittaire View Post
VMAF is objective metric from Netflix:
https://medium.com/netflix-techblog/...c-653f208b9652

Really good idea to impliment this metric. VMAF seem have really good correlation with HVS. Why not include PSNR-HVS-M (seem have really good correlation too)?

Possible to introduce tune profil for VMAF or PSNR-HVS-M like for PSNR and SSIM?
VMAF is our least-bad metrics, but it definitely has limitations and blind spots. It doesn't catch banding well, particularly in lower luma. It is only trained on x264 8-bit SDR, and without a lot of different psychovisual tuning, so it doesn't do a great job of discriminating between some kinds of psychovisual tradeoffs. And it's never seen the newer kind of artifacts in HEVC, so it doesn't know about customer preference between them. It doesn't know about <300 Kbps or >1080p. It was only tested against a particular display at 3x screen heights. The temporal comparison base metric is pretty weak.

It's the best thing we've got, but it definitely isn't perfect.

Also, the question of how to go from scores for individual frame to a per-clip score remains unanswered. A file that varies from 30-80 VMAF is going to be perceived as a lot worse than one that varies from 50-60, but the mean of the frames won't show any difference.

That's not a VMAF specific problem; it applies to all metrics where we are trying to figure out quality over time from individual frame metrics.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Instant Video

My Compression Book

Amazon Instant Video is hiring! PM me if you're interested.
benwaggoner is offline   Reply With Quote
Old 23rd May 2018, 08:32   #6146  |  Link
jd17
Registered User
 
Join Date: Jun 2017
Posts: 68
Wait a minute.... The Amazon video specialist does NOT think Netflix's metric is the holy grail?
I'm shocked! Shocked!
Kidding of course... I just think someone had to say it.

I do think Netflix abuses VMAF findings though.
Bitrates between 2000kbit/s and 3000kbit/s for 1080p x264 video are just not good. No matter what VMAF tells you...
jd17 is offline   Reply With Quote
Old 23rd May 2018, 08:39   #6147  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 5,309
The main limit is probably a bandwidth cap; restricted VBR. Some scenes will have a demand that can't be satisfied in a streaming usage. But quality tests will often be done for a local playback usage, I guess...
__________________

German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 23rd May 2018, 09:11   #6148  |  Link
jd17
Registered User
 
Join Date: Jun 2017
Posts: 68
Quote:
Originally Posted by LigH View Post
The main limit is probably a bandwidth cap; restricted VBR. Some scenes will have a demand that can't be satisfied in a streaming usage.
Is this a comment on my complaint about the 2000-3000kbit/s?
If that is the case - I do not think bandwidth cap is a factor.
I was referring to the "highest" quality Netflix streams at 1080p for some content. You cannot get more, no matter how high your bandwidth is.
This not only used to be much higher in the past, it still is for some content with more grain.
Additionally, Amazon streams significantly higher 1080p AVC bitrates across the board, so bandwidth can hardly be a limitation...

I think it's just Netflix saving bandwidth, because most people either watch this stuff on smaller screens with a viewing distance too large, or they just do not care / do not see the flaws...
Either way - Netflix abuses it.

Last edited by jd17; 23rd May 2018 at 09:17.
jd17 is offline   Reply With Quote
Old 23rd May 2018, 09:22   #6149  |  Link
nevcairiel
Registered Developer
 
Join Date: Mar 2010
Location: Hamburg/Germany
Posts: 9,241
The bitrate you target is really not related to the metric you use. You can use a metric to optimize encoding settings for any target bitrate to get the best out of the bits available to you.
__________________
LAV Filters - open source ffmpeg based media splitter and decoders
nevcairiel is offline   Reply With Quote
Old 23rd May 2018, 10:20   #6150  |  Link
jd17
Registered User
 
Join Date: Jun 2017
Posts: 68
You might be misunderstanding what I tried to say...
I know how metrics work.

I merely tried to point out that I think Netflix is abusing their internal VMAF findings to justify streaming at much lower bitrates.
I would argue they lowered their bitrate target (at least partly) as a result of their use of VMAF.

No matter how clean your video is, 1080p x264 will not be good at 2000-3000kbit/s, especially in the backgrounds.

Maybe you understand where I am coming from when you look at this blog:
https://medium.com/netflix-techblog/...n-7e99442b62a2

In theory, the arguments in that blog are well reasoned.
Use low bitrates for cartoons and clean video, use high bitrates for lots of movement, grain, and so on...

However, especially within the last year or so (look at recent releases of Netflix series for instance), they have moved to the aforementioned low bitrates as highest 1080p target bitrate. On live action video!

While 2000-3000kbit/s is indeed completely fine for Cartoon, it is not for live action video.

Is that VMAF's fault? Of course not.
But I think Netflix saw that the VMAF scores are not too bad for 1080p video at those bitrates, so they decided to lower their targets accordingly.


Sorry for the long off-topic discussion...
Originally, I just wanted to make a little joke and complain about Netflix's quality in the process...
I did not mean for it to drift so far away from x265!
jd17 is offline   Reply With Quote
Old Yesterday, 03:01   #6151  |  Link
The Stilt
Registered User
 
Join Date: May 2018
Posts: 2
Quote:
Originally Posted by NikosD View Post
I can make your life easy.

You have no choice. Nothing. Nothing at all.

Between 7900x and 1950x you can only buy one processor and that is of course 1950x or 2950x that will be released at Q3 2018.

The overall difference, as you already know, is huge favoring 1950x especially on rendering/ encoding and AVX512 is nonsense, pure nonsense for almost any task for desktop or even workstation.

You need 6 channel or 8 channel DDR4 or maybe DDR5 (2019/2020) to leverage effectively AVX512 and water cooling in most cases.

Let's forget AVX512 for the next 5 years for desktop and workstation.
Unless we're talking strictly about encoding material with > 1080 vertical resolution or running multiple encoder instances simultaneously, Threadripper is is pretty much the worst possible choice for HEVC encoding you can currently get.

Skylake-X has > 50% IPC advantage in the most recent encoder version over Ryzen, due to AVX2 (+41.7%) and AVX512 (+6.1%). Obviously Skylake-X is expensive and the power consumption is rather brutal, but regardless for typical end-user scenarios you're better basically with anything else than Threadripper. On a budget used MCC or HCC Haswell-EP CPUs are probably the best way to go.
The Stilt is offline   Reply With Quote
Old Yesterday, 06:45   #6152  |  Link
user1085
Registered User
 
Join Date: Apr 2018
Posts: 9
Quote:
Originally Posted by The Stilt View Post

Skylake-X has > 50% IPC advantage in the most recent encoder version over Ryzen, due to AVX2 (+41.7%) and AVX512 (+6.1%). Obviously Skylake-X is expensive and the power consumption is rather brutal, but regardless for typical end-user scenarios you're better basically with anything else than Threadripper. On a budget used MCC or HCC Haswell-EP CPUs are probably the best way to go.
I'm not seeing these gains with my AVX512 testing. Can you check if I have the right CLI options?

https://forum.doom9.org/showthread.php?t=175467
user1085 is offline   Reply With Quote
Old Yesterday, 17:57   #6153  |  Link
NikosD
Registered User
 
Join Date: Aug 2010
Location: Athens, Greece
Posts: 2,338
Quote:
Originally Posted by The Stilt View Post
Unless we're talking strictly about encoding material with > 1080 vertical resolution or running multiple encoder instances simultaneously, Threadripper is is pretty much the worst possible choice for HEVC encoding you can currently get.

Skylake-X has > 50% IPC advantage in the most recent encoder version over Ryzen, due to AVX2 (+41.7%) and AVX512 (+6.1%). Obviously Skylake-X is expensive and the power consumption is rather brutal, but regardless for typical end-user scenarios you're better basically with anything else than Threadripper. On a budget used MCC or HCC Haswell-EP CPUs are probably the best way to go.
There are no such things you describe, like IPC advantage or single instance encoding or even encoding below 1080p .

You seem desperate to prove Threadripper is lame compared to Skylake-X, by manipulating the user conditions.
Do you really think that the IPC advantage of a CPU has anything to do with a 10C/20T vs 16C/32T comparison ?

At the end of the day we are comparing final products and not just architectures at very specific situations.

Because those two CPUs have the same price.

Skylake-X is a choice for Intel's fanboys or for people who can afford burn money in the fireplace during the winter.
__________________
Win 10 x64 (17134.1) - Core i3-4170/ iGPU HD 4400 (v.4835)
HEVC decoding benchmarks
H.264 DXVA Benchmarks for all
NikosD is offline   Reply With Quote
Old Yesterday, 19:02   #6154  |  Link
The Stilt
Registered User
 
Join Date: May 2018
Posts: 2
Quote:
Originally Posted by NikosD View Post
There are no such things you describe, like IPC advantage or single instance encoding or even encoding below 1080p .

You seem desperate to prove Threadripper is lame compared to Skylake-X, by manipulating the user conditions.
Do you really think that the IPC advantage of a CPU has anything to do with a 10C/20T vs 16C/32T comparison ?

At the end of the day we are comparing final products and not just architectures at very specific situations.

Because those two CPUs have the same price.

Skylake-X is a choice for Intel's fanboys or for people who can afford burn money in the fireplace during the winter.
"There are no such things" can you elaborate a bit on that?

I have both, the 7960X and 1950X.
Even if I had only the 7900X, I'd still use it for HEVC encoding rather than the 1950X due to speed and higher efficiency.





Obviously in other multithreaded workloads the 1950X will annihilate the 7900X, but that's not the case with X265.

Quote:
Originally Posted by user1085 View Post
I'm not seeing these gains with my AVX512 testing. Can you check if I have the right CLI options?

https://forum.doom9.org/showthread.php?t=175467
Have you monitored the average clock frequency of the CPU cores between AVX2 and AVX512 options? At least on my 7960X the current AVX512 implementation in X265 does not trigger the AVX512 clock offset, however I'm not certain if the rules are the same for Xeons as well. I'd suggest you limit the encoder threads to a lower number (e.g. 8, --pools "8") and test again. At 205W power limit and 24 cores you are definitely hitting the power limits, which causes high variation in the frequency. Also, try using a raw YUV input (file) for the encoder, instead of piping the video from ffmpeg.
The Stilt is offline   Reply With Quote
Old Yesterday, 22:38   #6155  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,418
Quote:
Originally Posted by The Stilt View Post
"There are no such things" can you elaborate a bit on that?

I have both, the 7960X and 1950X.
Even if I had only the 7900X, I'd still use it for HEVC encoding rather than the 1950X due to speed and higher efficiency.
Annand tech bench is simply stupid and ridiculous test:
- 4K HEVC source at very high bitrate (~140 fps at 100% CPU charge just for stream decoding on i9-7900X and 1950X ... and Intel is really better than AMD for that)
- Really high speed setting (40 fps and more for 4K x265 encoding ... !!!?).

Certainely that Handbrake use high CPU charge just for 4K HEVC decoding source.

I make more serious bench here with 4K source and x265 encoding with preset "medium":
https://forum.doom9.org/showthread.php?t=174393

Code:
|----------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|--------|
| CPU            |   x264  |   x265  |   LAVC  |  auto   |   MMX2  |   SSE   |   SSE2  |   SSE3  |  SSE4   |   AVX   |   AVX2  |   All  |
|----------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|--------|
|  1950X         |  37.59  |   6.50  |    136  |   4.65  |   2.02  |   2.00  |   2.95  |   3.10  |   3.89  |   4.10  |   4.26  |   N/A  |
|  Core i9-7900X |  29.37  |   5.47  |    145  |   4.38  |   1.35  |   1.35  |   2.14  |   2.32  |   3.49  |   3.50  |   4.30  |   N/A  |
For real x265 encoding (less than 5% on CPU charge for source decoding):
- 1950x@stock 16C/32T is faster than i9-7900x@stock 10C/20T for x265 4K encoding.
- SSE4 at same speed and same core number is on par for Intel and AMD for x265 encoding.
- AVX2 produce 15-20% speed improuvement for Intel vs AMD in x265 encoding (on all modern CPU)
- AVX2 is not really efficient for x264: at same speed and same core number Intel and AMD are on par for speed encoding.
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9

Last edited by Sagittaire; Yesterday at 22:56.
Sagittaire is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 00:08.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.