Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th January 2019, 15:39   #21  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,717
Quote:
Originally Posted by Asmodian View Post
x265, with its 2x larger block sizes, actually hits that limit earlier. For 1080p I run two encodes at once on my i9 7900X.
Yes, there's a clear difference between CTU 64 and 32 what comes to parallel processing in the encoder. Of course, if you do extensive processing in Avisynth or better yet, Vapoursynth before feeding data into the encoder, the general utilization level should be higher by default.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 20th January 2019, 15:51   #22  |  Link
asarian
Registered User
 
Join Date: May 2005
Posts: 1,462
^^ Thx, guys. Good to know.
__________________
Gorgeous, delicious, deculture!
asarian is offline   Reply With Quote
Old 20th January 2019, 15:56   #23  |  Link
asarian
Registered User
 
Join Date: May 2005
Posts: 1,462
Quote:
Originally Posted by Asmodian View Post
x265, with its 2x larger block sizes, actually hits that limit earlier. For 1080p I run two encodes at once on my i9 7900X.
Why?! When I do heavy denoising stuffz, vspipe takes ca. 11-22% of the CPU. So, from that point of view, I can see why you would try and run multiple of those. The rest of the CPU is then usurped by the x264/x265 process.

So, what I don't understand then is how running two x265 processes concurrently lowers the amount of threads each uses. Or is that something you can actually set?

Thanks.
__________________
Gorgeous, delicious, deculture!
asarian is offline   Reply With Quote
Old 20th January 2019, 16:45   #24  |  Link
mparade
Registered User
 
Join Date: Nov 2013
Posts: 577
Quote:
Originally Posted by asarian View Post
On that matter -- if allowed here -- back in the day, for x264, it was said too many cores (= threads) would deteriorate the quality. Wasn't particularly relevant at the time, but nowadays (I'm thinking of getting an i9 9980XE myself, with 18 cores) the question may actually become pertinent again.

So, is this still an issue for x265? And, if so, at how many cores should we start to worry?
I am not using x265 for 1080p content just because x264 is able to provide the quality I want in the size interval I am aiming at keeping menus as well and very fast using my processor even with preset slower, however, I have to divide the content into chunks for a better processor utilization to reach.
For HD and SD res. I maximalize threads and lookahead threads to 12&4 and 6&2 correspondingly.
mparade is offline   Reply With Quote
Old 20th January 2019, 16:55   #25  |  Link
asarian
Registered User
 
Join Date: May 2005
Posts: 1,462
Quote:
Originally Posted by mparade View Post
I am not using x265 for 1080p content just because x264 is able to provide the quality I want in the size interval I am aiming at keeping menus as well and very fast using my processor even with preset slower, however, I have to divide the content into chunks for a better processor utilization to reach.
For HD and SD res. I maximalize threads and lookahead threads to 12&4 and 6&2 correspondingly.
Okay, thx. I'll look into that.
__________________
Gorgeous, delicious, deculture!
asarian is offline   Reply With Quote
Old 20th January 2019, 18:57   #26  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by Asmodian View Post
x265, with its 2x larger block sizes, actually hits that limit earlier. For 1080p I run two encodes at once on my i9 7900X.
Wait, you mean the devs of x265 didn't make its threading resolution aware as to not have it degrade image quality by spawning excess threads (by default)? I know x264 has that problem, but I thought I read that x265 was smarter about threading...
Stereodude is offline   Reply With Quote
Old 21st January 2019, 03:10   #27  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 4,403
Quote:
Originally Posted by asarian View Post
So, what I don't understand then is how running two x265 processes concurrently lowers the amount of threads each uses. Or is that something you can actually set?
It is something you can set with --pools and/or --frame-threads.

Quote:
Originally Posted by Stereodude View Post
Wait, you mean the devs of x265 didn't make its threading resolution aware as to not have it degrade image quality by spawning excess threads (by default)? I know x264 has that problem, but I thought I read that x265 was smarter about threading...
It is smarter, in that WPP helps, but it still allocates a thread for each logical core. By default it will also start using more frame threads, which hurt quality, as core count goes up.

"The number of frame threads used is auto-detected from the (hyperthreaded) CPU core count"
Cores: Frames
> 32: 6..8
>=16: 5
>= 8: 3
>= 4: 2
Source
__________________
madVR options explained
Asmodian is offline   Reply With Quote
Old 21st January 2019, 13:07   #28  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by Asmodian View Post
It is something you can set with --pools and/or --frame-threads.



It is smarter, in that WPP helps, but it still allocates a thread for each logical core. By default it will also start using more frame threads, which hurt quality, as core count goes up.

"The number of frame threads used is auto-detected from the (hyperthreaded) CPU core count"
Cores: Frames
> 32: 6..8
>=16: 5
>= 8: 3
>= 4: 2
Source
I've read that before. Maybe I'm misunderstanding it, but what I get out of that is that x265 will spawn a lot of threads on a high core count system, but they won't degrade image quality because they may largely sit idle waiting. They can impact determinism. Also, don't bother manually increasing the threading.

There certainly is no warning (or even a vague suggestion for that matter) to limit threading manually based on the input resolution for the sake of quality. In fact there's pretty much a statement to the opposite.
Quote:
When frame threading is disabled, the entirety of all reference frames are always fully available (by definition) and thus the available pixel area is not restricted at all, and this can sometimes improve compression efficiency. Because of this, the output of encodes with frame parallelism disabled will not match the output of encodes with frame parallelism enabled; but when enabled the number of frame threads should have no effect on the output bitstream except when using ABR or VBV rate control or noise reduction.
Stereodude is offline   Reply With Quote
Old 21st January 2019, 19:21   #29  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 4,403
Quote:
When frame threading is disabled, the entirety of all reference frames are always fully available (by definition) and thus the available pixel area is not restricted at all, and this can sometimes improve compression efficiency. Because of this, the output of encodes with frame parallelism disabled will not match the output of encodes with frame parallelism enabled; but when enabled the number of frame threads should have no effect on the output bitstream except when using ABR or VBV rate control or noise reduction.
What I understand from this is that restricting the number of threads in a pool is pointless when running one encode but setting --frame-threads 1 can improve compression efficiency. I believe "no effect on the output bitstream" means all the frames are the same types and it can be decoded the same way, not that there is no loss of compression efficiency.
__________________
madVR options explained
Asmodian is offline   Reply With Quote
Old 21st January 2019, 20:49   #30  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by Asmodian View Post
What I understand from this is that restricting the number of threads in a pool is pointless when running one encode but setting --frame-threads 1 can improve compression efficiency. I believe "no effect on the output bitstream" means all the frames are the same types and it can be decoded the same way, not that there is no loss of compression efficiency.
I took "no effect on the output bitstream" to mean no loss in visual quality because the encode is still deterministic (unless you use things that inherently break determinism).
Stereodude is offline   Reply With Quote
Old 21st January 2019, 22:36   #31  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 4,403
The first sentence in your quote doesn't make any sense then.

It cannot both sometimes improve quality when disabled and not impact quality when enabled. I think the second sentence is contrasting frame level parallelism against slices as a way to allow parallelism, slices change the bitstream.
__________________
madVR options explained

Last edited by Asmodian; 21st January 2019 at 22:40.
Asmodian is offline   Reply With Quote
Old 22nd January 2019, 03:39   #32  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by Asmodian View Post
The first sentence in your quote doesn't make any sense then.

It cannot both sometimes improve quality when disabled and not impact quality when enabled. I think the second sentence is contrasting frame level parallelism against slices as a way to allow parallelism, slices change the bitstream.
Well, I didn't write it. It's ambiguous on both ends.
Stereodude is offline   Reply With Quote
Old 22nd January 2019, 15:55   #33  |  Link
asarian
Registered User
 
Join Date: May 2005
Posts: 1,462
Quote:
Originally Posted by Asmodian View Post
The first sentence in your quote doesn't make any sense then.

It cannot both sometimes improve quality when disabled and not impact quality when enabled.
Why not?! Leaving a feature disabled can sometimes improve quality; and sometimes having the feature enabled will not impact quality.
__________________
Gorgeous, delicious, deculture!
asarian is offline   Reply With Quote
Old 22nd January 2019, 22:24   #34  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 4,403
Quote:
Originally Posted by asarian View Post
Why not?! Leaving a feature disabled can sometimes improve quality; and sometimes having the feature enabled will not impact quality.
For one, the second sentence does not say sometimes.

It also does not make sense technically. If you do not wait for all previous frames to be encoded sometimes you cannot chose what would have been the best reference block because it isn't available yet.

Quote:
When frame threading is disabled, the entirety of all reference frames are always fully available (by definition) and thus the available pixel area is not restricted at all, and this can sometimes improve compression efficiency. Because of this, the output of encodes with frame parallelism disabled will not match the output of encodes with frame parallelism enabled; but when enabled the number of frame threads should have no effect on the output bitstream except when using ABR or VBV rate control or noise reduction.
Also, what would the bold mean if "no effect on the output bitstream" means "no effect on quality"?

Edit: Maybe it is saying that once --frame-threads = 2 more frame threads should have no effect? I find this hard to believe, because more incomplete frames seems like it has to mean fewer reference blocks available, but it might be what is meant and I don't know how the details of reference block selection interacts with frame parallelism.
__________________
madVR options explained

Last edited by Asmodian; 22nd January 2019 at 22:40.
Asmodian is offline   Reply With Quote
Old 22nd January 2019, 22:34   #35  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,738
I read it as saying that frame-threads on/off can make a difference, but if frame threading is being used, the number of threads being used doesn't impact output quality.

This is not true with different number of frame-threads (with quality going down as the number goes up, although not nearly the impact it was in prior years). It might be true when frame-threads is constant but the number of threads in general varies.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 22nd January 2019, 22:51   #36  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 4,403
But it explicitly says "the number of frame threads should have no effect on the output bitstream" which really doesn't seem like it could be true if it means size v.s. quality.

Oh wait, it means the number of threads working on each frame, not the number of frames being worked on at once?
__________________
madVR options explained
Asmodian is offline   Reply With Quote
Old 23rd January 2019, 01:03   #37  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,738
Quote:
Originally Posted by Asmodian View Post
But it explicitly says "the number of frame threads should have no effect on the output bitstream" which really doesn't seem like it could be true if it means size v.s. quality.

Oh wait, it means the number of threads working on each frame, not the number of frames being worked on at once?
That is the only thing I thought it could mean that would make sense. The number of threads per frame go up with frame size. You can get quite a lot of cores going at once at UHD resolutions with high quality settings.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 6th March 2019, 16:54   #38  |  Link
jlw_4049
Registered User
 
Join Date: Sep 2018
Posts: 391
Wow, last I checked no one had responded to this. I appreciate all the information here!! Thanks!
jlw_4049 is offline   Reply With Quote
Old 7th March 2019, 23:35   #39  |  Link
blublub
Registered User
 
Join Date: Jan 2015
Posts: 118
Quote:
Originally Posted by jlw_4049 View Post
Wow, last I checked no one had responded to this. I appreciate all the information here!! Thanks!
lol.

Here is what I use:

1080P for high quality:
CRF: 17 to 19 - depending on the movie type / image noise
no-sao
frame-threads=3 to 4
ctu=32
merange=27
me=3
subme=4
rd=4
deblock=-1;-1
no-strong-intra-smoothing
bframes=4
ref=4
psy-rd=3.0
aq-mode=2 / aq=3 for dark movies
level-idc=51
high-tier
qcomp=0.63
vbv-maxrate=75000
vbv-bufsize=75000

4K HDR:
CRF 17 to 19 - same as above
--profile main10
--output-depth 10
--no-sao --frame-threads=3
--pme
--hdr-opt
--colorprim bt2020
--ctu=32
--qcomp=0.7
--no-strong-intra-smoothing
--aq-mode=2
--deblock=-1:-1
--level-idc=51
--high-tier
--qg-size=16
blublub is offline   Reply With Quote
Old 26th April 2019, 07:31   #40  |  Link
Nico8583
Registered User
 
Join Date: Jan 2010
Location: France
Posts: 851
Hi,
I tried some settings and some encodes but I can see every time x265 lose a lot of details, even with a high bitrate (2 pass 35 MBits, CRF16...)
Do you have some tricks to encode ?
Thank you !
Nico8583 is offline   Reply With Quote
Reply

Tags
handbrake, hevc, x265

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 12:40.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.