Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
6th December 2011, 12:42 | #21 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
6th December 2011, 12:45 | #22 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
*SLAPS OWN FOREHEAD* Of course.
Can the StackHorizontal be placed inside MP_Pipeline call, perhaps as a third process? No, again, we'd still need some way to represent the output of each of the other two processes to feed them into StackHorizontal. How about this: Code:
# Let's assume that all plugins are auto-loaded, for simplicity. AVIsource("some_anaglyph_3D_thing.avi") StackHorizontal( \ MP_Pipeline("""ExtractOneSide(Eye="Right")""", \ MP_Pipeline("""ExtractOneSide(Eye="Left")""" \ ) Is the use of the seperator (i.e. "### ###") mandatory to cause a new process to be created? Will adding a separator to one-line script (or extra separators in the general case) confuse the plugin, or will it just disregard any surplus ones? Can see I'm gonna need to have a play with this one, as soon as I get time. |
6th December 2011, 12:53 | #23 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
Code:
StackHorizontal( \ MP_Pipeline(""" AVIsource("some_anaglyph_3D_thing.avi") ExtractOneSide(Eye="Right") ### ### """, \ MP_Pipeline(""" AVIsource("some_anaglyph_3D_thing.avi") ExtractOneSide(Eye="Left") ### ### """ \ ) Yes, the "### ###" splitter is required to tell the plugin to spawn a new process. Script body can be empty though.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 Last edited by SAPikachu; 6th December 2011 at 13:03. |
|
6th December 2011, 12:54 | #24 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
SAPikachu: Heh, we posted almost simultaneously. Thanks for doing this, and for looking at ways to improve it. I think, if it can spawn a single processing chain (a single "mini-script") as a process separate from the calling script, then that's enough for me (see my revised example above).
But it would be cool also to provide a generalised framework for capturing the outputs of several processes: Perhaps add an "and finally..." section to MP_Pipeline's script template (with a different separator from the "start a separate process" one, and use generalised variables such as "Result1" "Result2" etc. to represent the outputs of the various processes, and which combines them into a single clip to pass back to the calling script...? Just brainstorming. Ignore me if I'm being thick. |
6th December 2011, 14:07 | #25 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
6th December 2011, 14:14 | #26 | Link | |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
Quote:
Yeah. Thinking about it earlier, I couldn't actually think of an example of how you'd use what I suggested that couldn't be done without it. That said, I'm thinking my StackHorizontal example isn't going to achieve any speed up anyway: StackHorizontal is in charge of which frames it calls for and when, and presumably waits until it has a complete frame from its first argument before asking for one from the second. So unless MP_Pipeline pre-requests frames from its sub-scripts, *ahead* of them being called for by StackHorizontal/the parent script, then there won't be any speed up. Does MP_Pipeline do that? |
|
6th December 2011, 15:38 | #27 | Link | |
Avisynth language lover
Join Date: Dec 2007
Location: Spain
Posts: 3,431
|
Quote:
Am I right in thinking that MP_Pipeline is essentially a source filter, and takes no clip input, or does it make use of 'last' in some way? |
|
6th December 2011, 23:54 | #28 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
Tried it out: Looks like you're right Gavino. MP_Pipeline.dll starts up each new process as an instance of MP_Pipeline.dll.slave.exe, which executes the relevant segment of the script via its own instance of AviSynth. Those processes don't appear (at present) to have any way of receiving input from the parent instance of AviSynth other than the script section itself. So, the AviSource (or equivalent) call has to be in the script section that's passed to MP_Pipeline.
A consequence of that is that if there's any common processing that needs to be done to the video before the processing paths diverge (e.g., in my usual cases, denoising and resizing the picture to fit half the screen width), that pre-processing will have to be run twice... Unless you prepare a mezzanine file first with a separate script. Common *post*-processing, on the other hand is easier: Just put it after the calls to MP_Pipeline. SAPikachu, can you confirm/critique that analysis? It works though! I did a test using an MVTools-based frame-doubler, as a simplified proxy for my 3D processing, and a dummy "minimal load encoder" - i.e. just cropping off most of the picture in VirtualDub and saving small rectangle of it, uncompressed. Using MP_Pipeline, instead of the same script without, finished in 58s rather than 95s. During processing, two processor cores were nearly fully used, rather than one. Last edited by pbristow; 7th December 2011 at 00:01. Reason: Forgot to say "It works!" :) |
7th December 2011, 05:11 | #29 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
In next version, I will add a script variable to slave AviSynth environment so that different slave process can be distinguished in script. And then you can use BRANCH statement to workaround this problem. But I think you can try ThreadRequest, in theory it can give bigger performance boost than MP_Pipeline, since overhead of multiprocessing is big.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 Last edited by SAPikachu; 7th December 2011 at 05:19. Reason: typo & some addendum |
|
9th December 2011, 04:07 | #30 | Link |
Registered User
Join Date: Aug 2007
Posts: 218
|
Released 0.2, forgot to reply yesterday.
@pbristow, you can try the new version like the snippet below, this script can eliminate duplicated preprocessing filters: Code:
MP_Pipeline(""" SomeSource() SomePreprocess() ### ### MP_PIPELINE_BRANCH_ID == 0 ? ExtractOneSide(Eye="Left") : ExtractOneSide(Eye="Right") SomeOtherProcess() Interleave(last, last) # important! ### branch: 2 ### ### LeftEye = SelectEvery(2, 0) RightEye = SelectEvery(2, 1) StackHorizontal(RightEye, LeftEye) """)
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 Last edited by SAPikachu; 9th December 2011 at 04:09. Reason: typo |
1st January 2012, 04:14 | #31 | Link |
Registered User
Join Date: Aug 2007
Posts: 218
|
Released 0.3. Thunked branching is implemented, but in my test it is pretty useless, since it usually decreases speed.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
11th February 2012, 14:11 | #32 | Link |
Registered User
Join Date: Aug 2007
Posts: 218
|
Released 0.9. I found a way to support multiple clip in thread-safe manner and implemented it in this version.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
3rd March 2012, 13:37 | #33 | Link |
Registered User
Join Date: Aug 2007
Posts: 218
|
Released 0.11, greatly improved performance.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
19th March 2012, 18:05 | #34 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
I've just been trying out this new version, and getting some astonishing results.
For example, I tried running an unmodified script and timeing the execution of a short job. Then I modified the script to split teh three main stages into separate threads within MP_Pipeline (i.e. I used three "### ###" separators). As expected, the CPU usage went up and execution time went dwn, but not by a huge amount as one of the three stages involves a lot more processing than the other two. Then I tried moving the later stages of processing out of teh MP_Pipeline call into the main script. This again was quicker than the original version, but not by much. ...But then I tried putting teh later stages back inside the MP_Pipeline call, *WITHOUT* the extra "### ###" separators, so that MP_Pipline was basically executing teh entire script in a single process, but separete from teh main AVIsynth process... ...and got teh fastest times of all! By a big margin! I've compared the output from the various techniques and they're all producing the exact same video, but for some reason invoking my entire script via MP_Pipeline is much quicker - *without* increasing the CPU usage noticably - than running it directly in AVisynth. I had trouble believing this, so I re-rane everything and timed them all again: Same result. Running the whole script via MP_Pipeline - with no other multithreading or multiprocessing being done - is quicker than any other method I've tried, using little or no extra CPU load than running the script directly. (Don't worry, I am remembering to add together the CPU sued by Virtualdub with that used by the MP_Piepline slave processes. ) Since nothing is apparently being "missed out" in the processing when I use MP_Pipeline, it follows that the normal, default mode of execution under avisynth is somehow being very wasteful of CPU cycles. Could this be a breakthrough in overall AVIsynth performance about to happen? N.B. I'm invoking AVIsynth via VirtualDub 1.10.0. It's *possible* the inefficiency is something to do with the communication between VirtualDub and Avisynth, but it seems unlikely. I'll try some more tests later on (after my work shift) with some different scripts, just in case it's an oddity of the particular plugins/filters I'm using, but in the meantime...Can anyone else confirm these results? |
19th March 2012, 18:18 | #35 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
OK, I just tried something on a hunch: I ran a simple script with just version() in it, first directly, then by wrapping an MP_Pipeline call round it.
Directly: AviSynth 2.58 MT v/6 (SVP edition), build: Oct 5 2010 [13:43:59] Via MP_Pipeline: AviSynth 2.58 tsp MT version 5(mod seraphy), build: Jul 12 2009 [07:46:21] MP_Pipeline is picking up a different installation of Avisynth to Virtualdub (one which seems to be twice as efficient!). How is it doing that?!? Turns out I have two avisynth.dll files active, one in system32 and one in sysWOW64. The one in sysWOW64 is version 2.5.8.6, is only sized 396KB and seems to be the one being picked up by VirtualDub. MP_Pipeline is presumably picking up the one in system 32 (version 2.5.8.5, size 3.44MB). Let's try removing the big fella from system32 (where it shouldn't be anyway)... Nope, that's not solved it. *WHERE* is MP_Pipeline getting its version of avisynth from? Last edited by pbristow; 19th March 2012 at 18:25. |
20th March 2012, 02:01 | #36 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
About the performance issue, can you post your tested scripts? By the way did you activate ### prefetch? That will also affect the speed.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
20th March 2012, 13:10 | #37 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
It turns out I had a copy of avisynth.dll sitting in the live AVIsynth *plugins* directory! MP_Pipeline was finding and using that, rather than the one in SysWOW64. After removing/renaming the one in plugins, MP_Pipeline picks up the same version as Virtualdub, and when I replaced that with the (more efficient) version I'd found in plugins, both Virtualdub and MP_Pipeline pick up the new version. So, everything is consistent now, I just need to settle on which version I should be running.
I'd suggest adding a note to your documentation for MP_Pipeline (if it isn't one there already) about the mechanism it uses to locate avisynth.dll, and the possible pitfalls if multiple versions exist on the system. Anyway, now I've got that sorted out, I'll start testing MP_Pipeline again. I have to say, so far it's been the least troublesome multi-processing technique I've tried, especially where a mix of temporal and spatial filters are being used and/or multiple sources files need parallel processing (e.g. to compare the results of multiple test runs). (BTW, No I wasn't using "### prefetch". Might try that next, after I've re-run the last batch of tests with my new, 100% less insane AVIsynth installation! ) |
20th March 2012, 13:30 | #38 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
Since you asked, here's my script, in the simplest version that uses MP_Pipeline. The other test scripts have the same content, just moving some of the later lines out of the "DoStuffToChunk" function to the main script.
The "Chunk" idea is a left-over from where I was trying to process the file in two parts. The idea is to use Interleave() to force the two chunks to be processed simultaneously store the result in a mezzanine file, and then witha second script separate the results out using SelectOdd and SelectEven and join the chunks together in the correct order. I might try that again now that I know my earlier results are contaminated... Code:
SetMemoryMax(96) # # Preamble: # SourceName = "TestClip.avi" AviSource(SourceName) TotFrames = FrameCount() Midpoint = TotFrames / 2 # # The Business: # function DoStuffToChunk(sourcename, start, end) { MP_Pipeline(""" SetMemoryMax(96) source = """" + sourcename + """" AviSource(source) Trim(""" + string(start) + """, """ + string(end) + """) SmartDecimate() ConvertToRGB24 Red = ShowRed() Blue = ShowBlue() Green = ShowGreen().GeneralConvolution(matrix = " 0 0 0 1 1 0 0 0 0 ") MergeRGB(Red,Green,Blue).ConvertToYV12 ### ### """) } ChunkA = DoStuffToChunk(sourcename, 1, TotFrames) Return ChunkA |
21st March 2012, 01:04 | #39 | Link |
soy sauce buyer
Join Date: Mar 2010
Location: United Kingdom
Posts: 164
|
Normally your VirtualDub test is getting YV12 data in the same thread of running avisynth script, while by using MP_Pipeline these two CPU costing tasks are separated into two threads - they are not competing for CPU resources in a single thread any more, and thus faster. If VirtualDub is not just running as video analysis or stream copy but with video encoding in the same time, the improvement in speed should be even more significant, unless your encoding process has already eaten up all the CPU resources without MPP.
|
21st March 2012, 01:12 | #40 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
06_taro, are you replying to me? If so, then yes, that's the idea of multiprocessing in general, and that's the result I was expecting to see: Shorter execution time with higher CPU use when using the NP_Pipeline plugin to split the workload between processes. The confusing results I got were due to my messed up multiple AVIsynth installations. Just one reason why it's always good to test these things rather than relying on the theory: Sometimes there's things going on that the theory didn't take into account.
|
Tags |
avisynth, multi-process, pipeline |
|
|