Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
21st March 2012, 03:10 | #41 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
So you have already found out that the performance issue was not due to the script. Just a suggestion, I think you can try using it together with ### branch and ### prefetch, that can make the script even more parallel and should have greater speed improvement.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
24th March 2012, 23:35 | #42 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
Hi SAPikachu,
can you explain a bit more about how branch and prefetch function, or how they should be used? I deduce from the MP_Pipeline_readme.avs script in the package that branch splits the video stream temporally (rather than spatially), like the traditional MT modes do, but in larger chunks (i.e. multiple frames per processing chunk). That (together with pre-fetch) should help with some temporal filters, i.e. the ones that get bogged down in duplicated effort under Set_MT_Mode: The only duplication will be around the boundaries of each chunk, so the bigger the chunk the better, yes? If you have several "### ###" process dividers, does it make any sense to use "### prefetch" in more than one place, or does one pre-fetch instruction apply to all processes? Continuing to test and play... |
25th March 2012, 01:33 | #43 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
prefetch statement only applies to the containing script block, so you usually need to add it to all script blocks. Be very careful if downstream script blocks have temporal filters, if prefetch cache size is not big enough, it may cause frequent seeking and cache thrashing, the speed will be greatly decreased because of that. Happy experimenting.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
25th March 2012, 13:54 | #44 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
Just checking my understanding again:
Code:
MP_Pipeline(""" AviSource("P:\Testfile.avi") ### ### InterFrame (FlowPath="C:\Program Files (x86)\AviSynth\Plugins\Dependencies\", Preset="Fast", NewNum=60000, NewDen=1001) ### prefetch: 20, 4 ### ### TDecimate(mode=0,cycleR=2,cycle=10) ### ### """) (P.S. Before anyone pipes up, I know that I'm using Interframe and TDecimate "the wrong way round". It's actually the best way for the job I'm doing! ) Hmm... This is turning into a usage rather than development thread. Do you want to move this discussion to the other forum? Last edited by pbristow; 25th March 2012 at 13:57. |
26th March 2012, 05:53 | #45 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
I think it's OK to keep the discussion here, since there is few development talk now.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
26th March 2012, 16:22 | #46 | Link |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
[QUOTE=SAPikachu;1567071]Yes, that's correct. Prefetch applies to the containing block. Frames from Interframe will be prefetched. Statement position in the script block doesn't matter though, even if you place the statement at the beginning, it will still be effective for the whole block.
QUOTE] I understand that it applies to the containing block; the question is, to which *end* of the block: The input, or the output? Pre-fetch is something that happens at the boundary between things, y'see. So is it pre-fetching *for* the current block, from the previous, or *from* the current block, for the next? |
27th March 2012, 13:52 | #47 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
[QUOTE=pbristow;1567133]
Quote:
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
28th March 2012, 15:53 | #48 | Link | |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
[QUOTE=SAPikachu;1567266]
Quote:
One thing: Audio doesn't seem to be carried through MP_Pipeline anymore. I'm having to re-dub the audio (using AudioDub(...) ) in the main script after the MP_Pipeline call, as none comes back from MP_Pipeline. Do you get the same, or am I doing something wrong? As far as I can see none of the filters I'm using should kill the audio, and if I move the AudioDub() call inside MP_Pipeline it doesn't work, but outside it does. |
|
29th March 2012, 01:22 | #49 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
30th March 2012, 00:25 | #50 | Link | |
Registered User
Join Date: Jun 2009
Location: UK
Posts: 263
|
Quote:
Overall impression: With the definite benefit from the prefetch facility (thanks for adding that! ), I'm finding it *most* useful in my comparison scripts, where I run the same basic pre-processing on several source files in parallel and then compare the results. Processing each source file via it's own slave process using MP_Pipeline and then combining the results in the main script works smoothly and efficiently every time, utilising up to 99% CPU and delivering speed gains to match. The only gotcha is memory: It's wise to use SetMemoryMax inside the MP_Pipleline call, with a parameter that won't overflow your free memory when all those parallel processes are demanding memory at once. (Formula: Take the average free memory (in Megabytes) you have when you first invoke VirtualDub (or whatever app calls AViSynth); divide that number by the number of parallel processes you're going to run; Shave off a few tens of megabytes for safety and/or uses that "SetMemoryMax" doesn't cover.) With more linear AviSynth scripts, the picture is not so straightforward. Sometimes using MP_Pipeline gives the best speed-up; sometimes SetMTmode() does. Sometimes tweaking the prefetch and branch parameters makes all the difference; sometimes it makes virtually none. Like all multi-threading and multiprocessing techniques, it really depends on knowing where and how to use it - and that means knowing the characteristics of the filters in your script and thus how they will be affected by the various different ways of dividing up the work they do - to get the best results. And sometimes, no matter what combination of tricks (MT_Pipline, MT(), SetMTMode(), etc...) and parameters I try, I can't get above 70% CPU utilisation, and that just has to be accepted. Some bottleneck in my machine prevents enough data getting to the CPU to be processed any faster, I guess. (Time to start saving up for that new motherboard...?) Big, big thanks to SAPikachu for a really useful plug-in. Last edited by pbristow; 30th March 2012 at 00:29. |
|
30th March 2012, 02:25 | #51 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
16th May 2012, 05:28 | #52 | Link |
Registered User
Join Date: Aug 2007
Posts: 218
|
Released 0.13. Fixed a crashing bug so it is recommended to update to this version.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
23rd December 2012, 03:26 | #53 | Link |
Registered User
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
|
I have avs 2.6 mt by SEt and avs64 2.5
and I use this script Code:
MP_Pipeline(""" ### platform: win64 DGDecode_MPEG2Source("x:\xx.d2v").ThreadRequest() SetMTMode(2) ColorMatrix(d2v="x:\xx.d2v", threads=0) Trim(x, xx) ++ Trim(xx, xxx) ++ Trim(xxxx, xxxxxx) SetMTMode(6) #~ tfm(output="matches.txt") #~ tdecimate(mode=4,output="metrics.txt") tfm(input="matches.txt") tdecimate(mode=5, hybrid=2, vfrDec=1, input="metrics.txt", tfmIn="matches.txt", mkvOut="mkv-timecodesfile.txt", tcfv1=false) Crop(x, x, -x, x) xxxResize(1280, 720) ### ### ### platform: win32 LoadPlugin(AviSynthPluginsDir + "EEDI2 mt/EEDI2_imp.dll") LoadPlugin(AviSynthPluginsDir + "avs26/mt_masktools-26.dll") SetMemoryMax(1000) SetMTMode(2) filter1 filter2 SoraThread() filter3 filter4 ### lock threads to cores ### ### """) AssumeFrameBased MP_Pipeline is cool plugin, I can use the script in x264 64 and x264 32, and I can use it in avspmod thank you for it ---------- edit 1: don't use MP_Pipeline in 1st pass (analysis) of tfm and tdecimate vfr (or animeivtc mode 4 with omode=2) Because it will generate empty files Last edited by real.finder; 14th May 2013 at 18:13. |
9th February 2013, 00:32 | #54 | Link |
Registered User
Join Date: Nov 2004
Location: Poland
Posts: 2,843
|
SAPikachu I have a problem with latest mp_pipeline. Avisynth 2.6mt and Vdub 32bit on Win 7 64bit.
When I load script to Vdub and than close it all slave processes stay active and take memory. I also think older version did not need loadPlugin....latest one does (or is it my imagination ) |
9th February 2013, 04:11 | #55 | Link | |
Registered User
Join Date: Aug 2007
Posts: 218
|
Quote:
About the slave process problem, I suspect avs-mt is the culprit. Can you try a regular avs build?
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
|
9th February 2013, 13:28 | #56 | Link |
Registered User
Join Date: Nov 2004
Location: Poland
Posts: 2,843
|
I fixed autoload by re-installing avisynth.
I tried "normal" avisynth and it's the same- processes don't close. I tried loading script to MP Classic and also the same I think older version was fine. |
11th February 2013, 12:28 | #57 | Link |
Registered User
Join Date: Aug 2007
Posts: 218
|
That's strange, it is fine on my machine. Can you try the following steps to capture a memory dump for me to debug?
1. Use this version of MP_Pipeline, and download and extract Procdump package 2. Load your script into vdub/mpc/whatever player, and close it 3. If the slave process is stuck, double-click "dump_immediately.cmd" in the Procdump package, and accept the agreement 4. A dump file (maybe very big) will be generated, please compress and upload it to any sharing site (Mediafire for example) and send it to me
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
13th February 2013, 08:45 | #59 | Link |
Registered User
Join Date: Aug 2007
Posts: 218
|
I just checked the dump, seems the slave process was stuck inside AppleProResDecoder, can you try disabling it and try again?
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18 ffms2 builds with 10bit output hack: libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1 Built from ffms2 6e0d654 (hack a9fe004) Mirrors: http://bit.ly/19TwDD3 |
13th February 2013, 13:53 | #60 | Link |
Registered User
Join Date: Nov 2004
Location: Poland
Posts: 2,843
|
I tried reading ProRes file with ffvideo and it was fine.
Than I tried on other machine and yes- with qtinput and ProRes file it's also a problem. I think it's qtinput+ProRes decoder- any way to solve it? Other mov formats seams to be fine and of course ProRes is the one which I'm interested the most Thanks for looking into it. |
Tags |
avisynth, multi-process, pipeline |
|
|