Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
25th May 2013, 09:25 | #81 | Link |
Registered User
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
|
for me using "lock threads to cores" make the process faster
and mp_pipeline very useful if you know how to use it correctly, Each script has a special case to deal with, and must be conduct experiments, and measurement speed by avs meter some time I use analysis pass in avspmod to measurement speed Good luck |
25th May 2013, 16:36 | #82 | Link | |
Registered User
Join Date: Sep 2012
Posts: 156
|
Quote:
Yes, thank you. That's a really useful statement given the current matters. edit: Evidently cpu's non-100% load is the problem here. I'm just trying a different script which went at 11fps on 1st pass and now it runs at 3.5fps with 40% cpu load. Last edited by aldix; 25th May 2013 at 16:42. |
|
27th May 2013, 02:54 | #83 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
I noticed you put many splitters (### ###) into your script, if you don't use prefetching it would be better to reduce count of splitters to roughly number of cpu cores you have. If you add too many splitters it may slow down your script. Then you can try moving splitters around to see if it can improve your speed.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
27th May 2013, 18:01 | #84 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
[SIGH] This is false goal. Forget it. The purpose of multi-tasking is to *speed up* the task you're working on, not to make your CPU as busy as possible. The only way you will ever get 100% utilisation on a multicore CPU is with extremely unusual (and probably artificial) workloads. Instead, the question to focus on is "how much time am I saving?" Here's a rough guide as to when to say "Woot!" rather than "Hmph! Well, that's better than nothing I suppose..." when using multi-tasking, versus the number of cores/threads you've used: Using 2: Execution time drops 30%. (fps increases 40%) Using 3: Execution time drops 42%. (fps increases 73%) Using 4: Execution time drops 50%. (fps increases 100%) Using 6: Execution time drops 59%. (fps increases 144%) Using 8: Execution time drops 66%. (fps increases 183%) ...In other words, don't expect a performance improvement much beyond the square root of the number of cores/processors/threads employed. |
27th May 2013, 18:22 | #85 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
...and, the post I'm replying to has already been deleted. [FACEPALM]
Perhaps I should make a couple of new rules for myself: (1) Don't reply to anything until it's at least a day old; (2) don't reply to anything without first updating the page six times to make sure no one else has already covered it! . |
27th May 2013, 21:18 | #86 | Link |
Registered User
Join Date: Sep 2012
Posts: 156
|
The only use for me that I can see is that big scripts wrapped into MP don't deadlock x264 anymore, even if not on full load at 2nd pass. Though that also was the case *before* all my recent MT troubles began (after updating to latest v).
So I don't really know what the take-away here is. Everything is just as slow for me as it previously was |
11th June 2013, 03:37 | #88 | Link |
Registered User
Join Date: Jul 2011
Posts: 1,121
|
I am trying to get MP Pipeline to work with MCTD and also work with audio.
So that it can work with Megui, making the audio bypass the MT Pipeline and be directly encoded. And the Video to be worked and processed in MCTD with MT Pipeline. But sadly, it doesnīt seem to work |
12th June 2013, 13:10 | #89 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
Maybe you can demux the audio and directly encode it, or simply comment out MP_Pipeline for your audio script. If you want to use the same script for both audio and video encoding, you can try AudioDub after MP_Pipeline.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
13th June 2013, 13:12 | #90 | Link |
Registered User
Join Date: Jul 2011
Posts: 1,121
|
Audiodub seems to work, for example:
u=Avisource("") MP_Pipeline(""" SetMemoryMax(3072) Avisource("") MCTD(settings="low") ### prefetch: 16, 0 ### ### """) AudioDub(u) So Thanks Though, i wonder is there a way to prevent MP to load when audio decoding? Cause currently, it seems to load MP which takes a very long time with more branches, and then they just close, and after all that, it starts processing. Last edited by zerowalker; 13th June 2013 at 13:16. |
14th June 2013, 01:29 | #91 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
I still think that you should demux your source audio and directly encode or mux it to the output, there is no reason to pass it through AviSynth unless you want to do some post-processing to it.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
15th June 2013, 02:08 | #93 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
CPU utilization mainly depends on whether works are fairly distributed across all script blocks, and how good is your prefetch setting. In other words, you need to make all the script blocks consume roughly same amount of CPU.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
15th June 2013, 18:15 | #94 | Link |
Registered User
Join Date: Sep 2012
Posts: 156
|
Well, I like smaller cpu load very much, it enables me to do other tasks. That's why I asked. I don't really understand prefetch, so I'm pretty much setting it at random, sometimes with what's suggested in the example script, sometimes way higher. I don't know...
|
15th June 2013, 21:59 | #95 | Link |
Registered User
Join Date: Jul 2011
Posts: 1,121
|
Okay, i think the audio goes pretty well now. Not sure why i went so slow before, must have done something wrong.
But i have another question. If i want to use MCTD, and run it in parallel in 4 threads on the same clip, how can i do that? I am thinking, that i should have 4 separate script which are identical, Except for the duration, meaning i have Trimmed the clip differently. And than later i can just suit them up. Though i have never done something like that so i am not absolutely sure of it. Or is it possible to have 1 script, with 4 branches, that each process a different "trim()" and MCTD? |
16th June 2013, 13:02 | #96 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
### prefetch controls how much frames ahead should be preloaded in background, and how many frames behind should be cached. Here is a simplified illustration of effect of prefetching: Code:
Without prefetching: T+0: +------------------+ | Upstream block | -> Producing frame #0 +------------------+ +------------------+ | Downstream block | -> Waiting for frame #0 from upstream +------------------+ T+1: +------------------+ | Upstream block | -> Wait for downstream block to complete processing frame #0 +------------------+ +------------------+ | Downstream block | -> Processing frame #0 +------------------+ T+2: +------------------+ | Upstream block | -> Producing frame #1 +------------------+ +------------------+ | Downstream block | -> Waiting for frame #1 from upstream +------------------+ T+3: +------------------+ | Upstream block | -> Wait for downstream block to complete processing frame #1 +------------------+ +------------------+ | Downstream block | -> Processing frame #1 +------------------+ --------------------------------------------------------------------------------- With prefetching: T+0: +------------------+ | Upstream block | -> Producing frame #0 +------------------+ +------------------+ | Downstream block | -> Waiting for frame #0 from upstream +------------------+ T+1: +------------------+ | Upstream block | -> Producing frame #1 +------------------+ +------------------+ | Downstream block | -> Processing frame #0 +------------------+ T+2: +------------------+ | Upstream block | -> Producing frame #2 +------------------+ +------------------+ | Downstream block | -> Processing frame #1 +------------------+ T+3: +------------------+ | Upstream block | -> Producing frame #3 +------------------+ +------------------+ | Downstream block | -> Processing frame #2 +------------------+
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
16th June 2013, 13:11 | #97 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
Splitting the script into parts and joining them after encoding may have better performance, but it may also affect rate-controlling of x264, making bitrate distribution worse.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
17th June 2013, 14:04 | #99 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
Code:
MP_Pipeline(""" XXXSource("...") ### prefetch: 512, 256 ### ### MCTD() ### branch: 4, 32 ### prefetch: 64, 0 ### ### ### prefetch: 64, 0 ### ### """) Actually, if you have only 8GB of memory, I would suggest trying this: Code:
MP_Pipeline(""" XXXSource("...") ### prefetch: 64, 32 ### ### SetMemoryMax(3072) # Set this to smaller value if your memory runs out or the slave process crashes MCTD() # Don't put anything else in this block ### prefetch: 32, 0 ### ### """)
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
17th June 2013, 14:17 | #100 | Link |
Registered User
Join Date: Jul 2011
Posts: 1,121
|
Okay, will be trying the second one.
And it seems to run at 9fps with my settings, which i guess is pretty good. Though i will have to Trim it still, as the Clip is 7 hours, the chance of it crashing is extremely high, it has happened so many times, that i have given up doing it in one run. |
Tags |
avisynth, multi-process, pipeline |
|
|