Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 13th July 2019, 03:36   #6901  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
AFAIK, piping can't seek either. The workaround is a giant pain. I have to make a unique .avs for every segment the outputs the right frames using trim, and then I have to make a unique .chp for every segment/x265 call that's shifted by the correct amount.
Stereodude is offline   Reply With Quote
Old 13th July 2019, 05:21   #6902  |  Link
alex1399
Registered User
 
Join Date: Jun 2018
Posts: 56
How about multiple piping? Just pipe the avs + lossless x264 encode ultrafast before the x265 ones
alex1399 is offline   Reply With Quote
Old 13th July 2019, 13:37   #6903  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by alex1399 View Post
How about multiple piping? Just pipe the avs + lossless x264 encode ultrafast before the x265 ones
Can you elaborate a bit more? I'm not sure I'm following you.
Stereodude is offline   Reply With Quote
Old 13th July 2019, 17:29   #6904  |  Link
alex1399
Registered User
 
Join Date: Jun 2018
Posts: 56
Quote:
Originally Posted by Stereodude View Post
Why hasn't direct AVIsynth input been added to the windows builds of x265 like x264 has? I'm rather sick of having to pipe video into it.
Never mind. Doesn't saw this previous post before.
alex1399 is offline   Reply With Quote
Old 14th July 2019, 04:27   #6905  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
Quote:
Originally Posted by Stereodude View Post
AFAIK, piping can't seek either. The workaround is a giant pain. I have to make a unique .avs for every segment the outputs the right frames using trim, and then I have to make a unique .chp for every segment/x265 call that's shifted by the correct amount.
Can you shed more light on what you are trying to accomplish?

Why would a chp file be affected by seeking support? Suppose it's a chapter file, and x265 does not take chapter files as input.

Talking about my experience, if I want to backup a movie, usually I want to encode it in multiple segments. I have scripts which can produce "import().trim()" type of AVS files, and I encode them in different batch, then concatenate them together, attach the chapter, and call it a day.
__________________
Projects
x265 - Yuuki-Asuna-mod Download / GitHub
TS - ADTS AAC Splitter | LATM AAC Splitter | BS4K-ASS
Neo AviSynth+ filters - F3KDB | FFT3D | DFTTest | MiniDeen | Temporal Median
MeteorRain is offline   Reply With Quote
Old 14th July 2019, 09:20   #6906  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
3-level Hierarchial Motion Estimation

I see there are two recent commits related to hierachial motion estimation. I couldn't find anything specific as to what the use cases would be, does anyone have any insights? Or is this once again one of those things that just appear in the encoder out of the blue (probably requested by some paying client) and the devs leave it at that.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 14th July 2019, 09:34   #6907  |  Link
microchip8
ffx264/ffhevc author
 
microchip8's Avatar
 
Join Date: May 2007
Location: /dev/video0
Posts: 1,843
Quote:
Originally Posted by Boulder View Post
I see there are two recent commits related to hierachial motion estimation. I couldn't find anything specific as to what the use cases would be, does anyone have any insights? Or is this once again one of those things that just appear in the encoder out of the blue (probably requested by some paying client) and the devs leave it at that.
Here is a rather technical explanation http://homepages.inf.ed.ac.uk/rbf/CV...stimation.html
__________________
ffx264 || ffhevc || ffxvid || microenc
microchip8 is offline   Reply With Quote
Old 14th July 2019, 11:58   #6908  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by MeteorRain View Post
Can you shed more light on what you are trying to accomplish?

Why would a chp file be affected by seeking support? Suppose it's a chapter file, and x265 does not take chapter files as input.

Talking about my experience, if I want to backup a movie, usually I want to encode it in multiple segments. I have scripts which can produce "import().trim()" type of AVS files, and I encode them in different batch, then concatenate them together, attach the chapter, and call it a day.
I'm encoding something in multiple segments to improve quality (less threads) and speed (multiple encodes at once). I'm talking about having to shift frames around in the qpfile. Each unique .avs needs a unique qpfile.

Seek support would simply this since the same .avs and qpfile could be used and only the seek point and frame number on the command line would need to change.

This would also be more simple if affinity restrictions from the START command in Windows 10 worked with x265. It seems that somewhere in the Windows 10 feature update chain MS changed the behavior. Affinity restrictions on a .bat file that called an executable used to result in the same affinity restrictions being placed on the executable started by the .bat file. Now it doesn't. Affinity restrictions placed directly on the x265 command line don't even work. (They do work on the x264 command line.)

Ideally I would run a batch file with these four lines and be done.
Code:
START "Enc #1" /NORMAL /NODE 0 /AFFINITY F000 "C:\HDTV Tools\x265\x265.exe" -F 1 --crf 16.0 -p veryslow ...
START "Enc #2" /NORMAL /NODE 0 /AFFINITY 0F00 "C:\HDTV Tools\x265\x265.exe" -F 1 --crf 16.0 -p veryslow ...
START "Enc #3" /NORMAL /NODE 0 /AFFINITY 00F0 "C:\HDTV Tools\x265\x265.exe" -F 1 --crf 16.0 -p veryslow ...
START "Enc #4" /NORMAL /NODE 0 /AFFINITY 000F "C:\HDTV Tools\x265\x265.exe" -F 1 --crf 16.0 -p veryslow ...
Each x265 instance would see only 4 logical cores and would be locked to those 4 cores. But instead each x265 instance sees all 16 logical cores and I have to add --pools 4 to the command line and then manually modify the affinity of each x265 process in the Task Manager once they're all started.
Stereodude is offline   Reply With Quote
Old 14th July 2019, 15:06   #6909  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Quote:
Originally Posted by froggy1 View Post
Here is a rather technical explanation http://homepages.inf.ed.ac.uk/rbf/CV...stimation.html
Thanks - yes, quite technical. To me it looks somehow similar to MRecalculate in MVTools, but maybe someone can explain it in layman's terms or simply come up with some POC they must have performed at some point. Looks interesting anyway.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 14th July 2019, 17:34   #6910  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
Quote:
Affinity restrictions placed directly on the x265 command line don't even work. (They do work on the x264 command line.)
I can confirm this behaviour. Looks like that x265 unlike x264 resets affinity at launch. I would also prefer that x265 didn't change initial affinity set by user.
Atak_Snajpera is offline   Reply With Quote
Old 15th July 2019, 12:01   #6911  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
I believe it may be possible to use the --pools parameter to control the affinity of x265 instances if it was just documented verbosely enough...

Quote:
x265 creates one or more thread pools per encoder, one pool per NUMA node (typically a CPU socket). --pools specifies the number of pools and the number of threads per pool the encoder will allocate. By default x265 allocates one thread per (hyperthreaded) CPU core on each NUMA node.
This may be not sufficient without examples. Right away you will notice that the online documentation does not even mention some parameters in the help output:

Code:
   --pools <integer,...>         Comma separated thread count per thread pool (pool per NUMA node)
                                 '-' implies no threads on node, '+' implies one thread per core on node
I would guess that calls with a similar result as the above mentioned "start /AFFINITY" might look similar to:

Code:
x265 --pools 4,"+---"
x265 --pools 4,"-+--"
x265 --pools 4,"--+-"
x265 --pools 4,"---+"
I hope Ma can enlighten us.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is online now   Reply With Quote
Old 15th July 2019, 12:12   #6912  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by LigH View Post
I believe it may be possible to use the --pools parameter to control the affinity of x265 instances if it was just documented verbosely enough...



This may be not sufficient without examples. Right away you will notice that the online documentation does not even mention some parameters in the help output:

Code:
   --pools <integer,...>         Comma separated thread count per thread pool (pool per NUMA node)
                                 '-' implies no threads on node, '+' implies one thread per core on node
I would guess that calls with a similar result as the above mentioned "start /AFFINITY" might look similar to:

Code:
x265 --pools 4,"+---"
x265 --pools 4,"-+--"
x265 --pools 4,"--+-"
x265 --pools 4,"---+"
I hope Ma can enlighten us.
Your example doesn't work. And, it contradicts what's in this: https://x265.readthedocs.io/en/default/cli.html

It has these examples:
Code:
""        - default, unspecified, all numa nodes are used for thread pools
"*"       - same as default
"none"    - no thread pools are created, only frame parallelism possible
"-"       - same as "none"
"10"      - allocate one pool, using up to 10 cores on all available nodes
"-,+"     - allocate one pool, using all cores on node 1
"+,-,+"   - allocate one pool, using only cores on nodes 0 and 2
"+,-,+,-" - allocate one pool, using only cores on nodes 0 and 2
"-,*"     - allocate one pool, using all cores on nodes 1, 2 and 3
"8,8,8,8" - allocate four pools with up to 8 threads in each pool
"8,+,+,+" - allocate two pools, the first with 8 threads on node 0, and the second with all cores on node 1,2,3
There's considerably more verbage on it too.

I see no references to core restrictions, only node restrictions in the documentation.

Last edited by Stereodude; 15th July 2019 at 12:16.
Stereodude is offline   Reply With Quote
Old 15th July 2019, 12:27   #6913  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
Ah, I missed these examples... I would have given a better guess if I knew them.

So it seems to be:

Code:
x265 --pools "4,4,4,4,+,-,-,-"
x265 --pools "4,4,4,4,-,+,-,-"
x265 --pools "4,4,4,4,-,-,+,-"
x265 --pools "4,4,4,4,-,-,-,+"
Or am I still wrong? ... In this case, I may misunderstand the nodes in contrast to the specific cores.

P.S.: I asked in the x265 developer mailing list.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid

Last edited by LigH; 15th July 2019 at 12:42.
LigH is online now   Reply With Quote
Old 15th July 2019, 13:30   #6914  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by LigH View Post
Ah, I missed these examples... I would have given a better guess if I knew them.

So it seems to be:

Code:
x265 --pools "4,4,4,4,+,-,-,-"
x265 --pools "4,4,4,4,-,+,-,-"
x265 --pools "4,4,4,4,-,-,+,-"
x265 --pools "4,4,4,4,-,-,-,+"
Or am I still wrong? ... In this case, I may misunderstand the nodes in contrast to the specific cores.

P.S.: I asked in the x265 developer mailing list.
I didn't try any of those, but this system only has 1 numa node (8C/16HT) and those command lines would be for a system with 4 numa nodes, and they're contradicting. "0,0,0,4" would put 0 thread pools on the first 3 Numa nodes and 4 on the 4th. "4,4,4,4" would put 4 thread pools on each of the 4 Numa nodes. "+,-,-,-" would put a thread pool with an number equal to the logical cores in the first node and none on the next 3. So, you've got conflicting instructions on the command line.

By my reading and messing with them (I do have a dual Numa node system) you have control over nodes, but not over cores.

I'll be curious to see what reply you get on the mailing list.
Stereodude is offline   Reply With Quote
Old 15th July 2019, 22:23   #6915  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
I have never made x265 to set its affinity as I intended so I always use my modded version of avs4x264 to do the job, which overwrites the affinity after x265 launches.

Regarding to your qpfile usage, honestly I have no similar usage of those so I don't have good suggestions for you. Usually if I want to force an IDR I'll just cut that as a new segment.

I'll take a quick look at the source code and see if I can find a way to do a seek easily. If it's easy enough to implement I'll slipstream it into my patch set.
__________________
Projects
x265 - Yuuki-Asuna-mod Download / GitHub
TS - ADTS AAC Splitter | LATM AAC Splitter | BS4K-ASS
Neo AviSynth+ filters - F3KDB | FFT3D | DFTTest | MiniDeen | Temporal Median
MeteorRain is offline   Reply With Quote
Old 15th July 2019, 23:50   #6916  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
And -- I might be wrong, but -- I don't see any code with qpfile that takes into account seek option. That means you'll have to shift your qpfile for any seek option you applied in the command line.
__________________
Projects
x265 - Yuuki-Asuna-mod Download / GitHub
TS - ADTS AAC Splitter | LATM AAC Splitter | BS4K-ASS
Neo AviSynth+ filters - F3KDB | FFT3D | DFTTest | MiniDeen | Temporal Median
MeteorRain is offline   Reply With Quote
Old 16th July 2019, 02:21   #6917  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
https://down.7086.in/x265-10bit-avisynth-seek-test.zip

qpfile still needs manual shifting.
__________________
Projects
x265 - Yuuki-Asuna-mod Download / GitHub
TS - ADTS AAC Splitter | LATM AAC Splitter | BS4K-ASS
Neo AviSynth+ filters - F3KDB | FFT3D | DFTTest | MiniDeen | Temporal Median
MeteorRain is offline   Reply With Quote
Old 16th July 2019, 02:33   #6918  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by MeteorRain View Post
And -- I might be wrong, but -- I don't see any code with qpfile that takes into account seek option. That means you'll have to shift your qpfile for any seek option you applied in the command line.
So if I tell it to drop an I frame at frame 6000 and I tell it to seek to frame 5000 to start the encode it doesn't subtract 5000 from 6000 and know to place it at 1000 in the encode (which is frame 6000 in the source)?

Stereodude is offline   Reply With Quote
Old 16th July 2019, 03:21   #6919  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
That is correct as far as I know.
__________________
Projects
x265 - Yuuki-Asuna-mod Download / GitHub
TS - ADTS AAC Splitter | LATM AAC Splitter | BS4K-ASS
Neo AviSynth+ filters - F3KDB | FFT3D | DFTTest | MiniDeen | Temporal Median
MeteorRain is offline   Reply With Quote
Old 16th July 2019, 23:25   #6920  |  Link
mandarinka
Registered User
 
mandarinka's Avatar
 
Join Date: Jan 2007
Posts: 729
Quote:
Originally Posted by Boulder View Post
I see there are two recent commits related to hierachial motion estimation. I couldn't find anything specific as to what the use cases would be, does anyone have any insights? Or is this once again one of those things that just appear in the encoder out of the blue (probably requested by some paying client) and the devs leave it at that.
If I'm not mistaken, this method could perhaps improve the search by allowing it to find more distant motion vectors (large motion difference between frames say in anime action sequences?). I'm completely assuming, haven't tried yet.

Has anybody here tested it?

(Edit: also, is it finished/ready for testing yet, or are there more incoming patches to plumb it together?)
mandarinka is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 20:11.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.