Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th January 2020, 01:57   #1  |  Link
sonyzz
Registered User
 
Join Date: Apr 2019
Posts: 3
high core count cpu for x265 encoding good idea?

as i already have x570 motherboard and ram bought like 4 months ago i'm planning to get either 3900x or 3950x from AMD which both are high core count cpu's and wanted to ask how much cores does x265 or hevc encoding can handle? i know that my used program can max out 8 core cpu (tested with friends ryzen 7 3700x) which speeds up video encoding... also another question would be - why do people on other topics say ''Note that quality diminishes as the number of cores/threads used to encode a video file increases. Once that number reaches the double digits, the quality decrease starts to become noticeable. '' why quality should decrease, if encode is being done twice faster because of more cores on same preset and same settings are people suggesting that high core count cpu cannot be used with x265 to produce good encoding results? thus i'm on BIG dilema here also please simplify the answers if possible thanks
sonyzz is offline   Reply With Quote
Old 26th January 2020, 12:33   #2  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,374
x265 using default settings won't saturate 3950x while encoding 1920x800 (2.4:1 aspect ratio) video.
However if you use Distributed Encoding in ripbot264 then you can fully utilize even dual AMD Epyc 2 (256 threads total) without any problems.

Last edited by Atak_Snajpera; 26th January 2020 at 12:45.
Atak_Snajpera is offline   Reply With Quote
Old 26th January 2020, 15:23   #3  |  Link
sonyzz
Registered User
 
Join Date: Apr 2019
Posts: 3
Quote:
Originally Posted by Atak_Snajpera View Post
x265 using default settings won't saturate 3950x while encoding 1920x800 (2.4:1 aspect ratio) video.
However if you use Distributed Encoding in ripbot264 then you can fully utilize even dual AMD Epyc 2 (256 threads total) without any problems.
But is it really as people say that if core/thread count increases - encoding quality decreases, and if it decreases does it decrease drastically? or for example like 1% from each added core so lets say for 4 core would be 5% loss for 8 would be 10% something like this, because now i was about to get atleast 3900x to make my encoding faster and to have spare resources when encoding is running, for example so i can encode and game at same time... because my 4c 8t cpu lets me encode and use the pc fine no problems (feels a bit less responsive when encoding and cpu is at 99%) and also can play games like lol with some ping or fps drops but its not good experience, AAA games are unplayable when encoding...
sonyzz is offline   Reply With Quote
Old 26th January 2020, 15:51   #4  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,374
That issue was totally overblown. x264 has a hard thread limit (24 threads if I remember correctly) to prevent that quality degradation. In x265 default large CTU (64) also works as limiting factor for thread utilization.

Last edited by Atak_Snajpera; 26th January 2020 at 15:58.
Atak_Snajpera is offline   Reply With Quote
Old 26th January 2020, 18:04   #5  |  Link
excellentswordfight
Lost my old account :(
 
Join Date: Jul 2017
Posts: 130
Quote:
Originally Posted by sonyzz View Post
But is it really as people say that if core/thread count increases - encoding quality decreases, and if it decreases does it decrease drastically? or for example like 1% from each added core so lets say for 4 core would be 5% loss for 8 would be 10% something like this, because now i was about to get atleast 3900x to make my encoding faster and to have spare resources when encoding is running, for example so i can encode and game at same time... because my 4c 8t cpu lets me encode and use the pc fine no problems (feels a bit less responsive when encoding and cpu is at 99%) and also can play games like lol with some ping or fps drops but its not good experience, AAA games are unplayable when encoding...
I woudlnt worry to much about quality degradation when it comes to threading as long as you stay within the more or less default values.

For 1080p video with preset slow you will see pretty good scaling up to 8c/16t, if you lower the CTU size to 32 or encode 2160p you will probably be good up towards 16C/32T. After that point doing chunk encoding is pretty much a must to continue thread scaling.
excellentswordfight is offline   Reply With Quote
Old 26th January 2020, 21:38   #6  |  Link
blublub
Registered User
 
Join Date: Jan 2015
Posts: 114
Hi

I have a TR3960x and I have learned the following:

Quality decreases in x265 if you you more frame-threads - auto setting is 6 for high core count CPUs.
For a UHD encode I can utilize about 80-90% of CPU usage with pool=64 - making it using 64 threads instead of auto 48.

Especially for 1080p it makes sense to either run 2 encodes at the same time or distribute the encoding via RipBot to use all CPU cores.
blublub is offline   Reply With Quote
Old 26th January 2020, 23:58   #7  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,439
Quote:
Originally Posted by Atak_Snajpera View Post
That issue was totally overblown. x264 has a hard thread limit (24 threads if I remember correctly) to prevent that quality degradation. In x265 default large CTU (64) also works as limiting factor for thread utilization.
Could ripbot264 be re-written to use ffmpeg as source/trimming/encoder, instead of avisynth?
kolak is offline   Reply With Quote
Old 27th January 2020, 09:53   #8  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,374
Quote:
Originally Posted by kolak View Post
Could ripbot264 be re-written to use ffmpeg as source/trimming/encoder, instead of avisynth?
Ffmpeg does not support frame accurate seeking. Basicaly you can't tell ffmpeg to seek to specific frame. On other hand lsmash plus avisynth is much more reliable and flexible. Ffmpeg only method would be huge downgrade in every aspect.
Atak_Snajpera is offline   Reply With Quote
Old 27th January 2020, 22:57   #9  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,439
If you let it decode source then it will be frame accurate. Similar to wait for index file. For I frame based codecs -ss before -i is also frame accurate.
You can also seek this way:
-ss 8 -i xxxx -ss 4.08 which will seek to 8th second as rough seeking and then 4.08 second from this place as precise seeking. This way you should be able to seek very quickly and precisely in any format if I understand it correctly.

avs/vs source filters are its weakest point and not 100% reliable at all. I ha many problems and had to choose correct one base don source format (eg. problem with transport streams or MXF files). They are actually based on ffmpeg and sometimes there are problems even after indexing.

Last edited by kolak; 27th January 2020 at 23:21.
kolak is offline   Reply With Quote
Old 27th January 2020, 23:20   #10  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,374
That's the problem. Seeking based on time is not precise enough. Trust me soon or latter you will end up with chunks starting from incorrect frame. I remember that mediacoder was trying this method and failed misserable. Check his forum and posts related with distibutive encoding.
Atak_Snajpera is offline   Reply With Quote
Old 27th January 2020, 23:23   #11  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,439
Only problem I found is sometimes for fractional frames (I assume due to rounding-it would need figuring out how ffmpeg does it). For non-fractional frames never had an issue. Eg. 0.04 is 1 frame for 25p based source. Quite simple.

If you happy to wait for decode time then you have very precise access with select filter. Then you can use frame numbers.

Besides, for distributed encoding we either pick "nice/round" numbers or scene changes. There is really no need for places like 00:10:12:11 (if it's not a scene change).
You can with ffmpeg jump to rough place, then from this point find 1st clear scene change and get time for it. For 2nd part you jump to same rough place and add time for scene change+ 1 frame (this should actually "force" ffmpeg to "use same math" for both chunks time points).
Actually this could work very well.

Last edited by kolak; 27th January 2020 at 23:36.
kolak is offline   Reply With Quote
Old 27th January 2020, 23:32   #12  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,374
I'm happy with l-smash (100% frame accurate) + avisynth so going back to ffmpeg only method would be like opening pandora box. Besides. If ain't broken, don't fix IT.
Atak_Snajpera is offline   Reply With Quote
Old 27th January 2020, 23:40   #13  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,439
If you think about it then it's 5% code change or rather 2nd version. It only changes what you execute, but whole main engine would stay the same.
I'm not programmer, so won't do it, but I realised it can be not so difficult even just with Python (quite easy for just 1 machine).
kolak is offline   Reply With Quote
Old 27th January 2020, 23:45   #14  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,374
But that constant seeking without index would significantly delay encoding for later chunks.
First chunk would start immediatelly, but how about 100th? Also keep in mind that everything is going VIA network and multiple servers would do the same task simultanously.
Atak_Snajpera is offline   Reply With Quote
Old 27th January 2020, 23:52   #15  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,439
Yes, could be the issue, but ss seeking should be enough.
I've tested it and it works even with ts files with h264 (which are pain to seek).

Last edited by kolak; 27th January 2020 at 23:54.
kolak is offline   Reply With Quote
Old 27th January 2020, 23:56   #16  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,374
Ha! Try with interlaced vc-1...
Btw. Why are so anti-avisynth?
Atak_Snajpera is offline   Reply With Quote
Old 28th January 2020, 00:03   #17  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,439
Not anti avs at all
I need as simple and reliable solution as possible. My workflows require 99.99% reliability.
I used avs for massive jobs (eg. 3000 hours of very heavy processing) and it did work, but ffmpeg is "easier" and multi platform as well.
Interlaced vc-1 - not interested in it at all. Should never exists
I deal with broadcast+intermediate formats, so XDCAM MXF, ProRes, DNxHD, AVC-I MXF etc. (mainly MXF and MOV).

Last edited by kolak; 28th January 2020 at 00:05.
kolak is offline   Reply With Quote
Old 28th January 2020, 00:10   #18  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,374
But I need to cover all those formats. Another problem. Without avisynth you would not have Access to mdegrain,qtgmc,hdrtosdr tonemspping or multithreaded resizer. I guess you do not also do any filtering like denoising, deinterlacing...
Atak_Snajpera is offline   Reply With Quote
Old 28th January 2020, 00:53   #19  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,439
Deinterlacing for sure. De-noising not really.
I actually use vs for deinterlacing. It only deinterlaces interlaced frames and uses QTGMC. Same can be done with ffmpeg very easily, just less good deinterlacing. Although recent new filter (bwdif= yadif+w3fdif) is good enough for broadcast and of course so much faster than QTGMC.
kolak is offline   Reply With Quote
Old 28th January 2020, 17:48   #20  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 689
For high core count cpu I'd reduce frame threads to 1 and pools to 12 and do chunked encoding (half-half) or parallel encoding (2 at a time).

I believe less threads = less waste on threading for x265, so I always use less threads and run more processes.

Besides, I already have the software infrastructure to do chunked encoding, and mux them later. Encoded GOPs can be joined later, so chunked encoding is basically zero cost to me.
MeteorRain is offline   Reply With Quote
Reply

Tags
3900x, 3950x, hevc, ryzen, x265

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 08:23.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.