Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
4th August 2020, 21:46 | #1 | Link |
Software Developer
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,248
|
Windows pipe weirdness
So, I have a command-line program that generates a stream of pseudo-random bytes (using simple "xorshift" method) and writes them to the STDOUT.
Now I wanted to measure the throughput (output rate) of the generator. On Linux, I would have used pv for that purpose, but I was on Windows and couldn't find a native Win32 port of pv. So, I ran the following command in an MSYS2 shell, where pv can be installed from the package manager: ./generator.exe | pv -br > /dev/null This resulted in a measured throughput of about ~1.2 GiB/s. Then I though I could write a "native" Win32 pv-like program with a few lines of code. So I did exactly that, and ran it from the Windows command prompt: generator.exe | native_pv.exe > NUL This time, however, I got a throughput of only ~500 MiB/s Interesting observation: The "native_pv.exe" process was constantly using 12.5% of the CPU (on a Quadcore with HT that is), whereas the "generator.exe" process was idling at ~5% CPU usage. Normally "generator.exe" uses 12.5% of the CPU, i.e. the maximum a single-threaded process can use. But what's really bugging me is that, if I run the following from within the MSYS2 shell, then I suddenly get the full ~1.2 GiB/s again: ./generator.exe | ./native_pv.exe > /dev/null This means that, apparently, not my "native_pv.exe" program sucks, but the pipe behaves quite differently in the MSYS2 shell than it does in the native Windows command prompt Any ideas what is going on here? (I would have expected that, if at all, the MSYS2 shell should be slower than the native command prompt, because it adds another emulation layer, i.e. msys-2.0.dll, on top of the Win32 API)
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊ Last edited by LoRd_MuldeR; 6th August 2020 at 17:14. |
5th August 2020, 07:32 | #2 | Link |
Registered Developer
Join Date: Mar 2010
Location: Hamburg/Germany
Posts: 10,348
|
Did you open the pipe in binary mode, to avoid any crazy overhead from Windows processing the stream in text mode?
__________________
LAV Filters - open source ffmpeg based media splitter and decoders |
5th August 2020, 15:43 | #3 | Link | |
Software Developer
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,248
|
Quote:
Anyway, today I create a little helper application that launches two sub-processes and connects them via pipe. Because, this way, the creation of the pipe is under my control, I can choose the buffer size. Turns out that this command, in the Windows command prompt, results in ~1.1 GiB/s: mkpipe.exe generator.exe "|" native_pv.exe > NUL (Pipe buffer size was set to 1 MiB)
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊ Last edited by LoRd_MuldeR; 6th August 2020 at 17:13. |
|
|
|