Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Programming and Hacking > Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 4th August 2020, 21:46   #1  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,146
Windows pipe weirdness

So, I have a command-line program that generates a stream of pseudo-random bytes (using simple "xorshift" method) and writes them to the STDOUT.

Now I wanted to measure the throughput (output rate) of the generator. On Linux, I would have used pv for that purpose, but I was on Windows and couldn't find a native Win32 port of pv.

So, I ran the following command in an MSYS2 shell, where pv can be installed from the package manager:
./generator.exe | pv -br > /dev/null

This resulted in a measured throughput of about ~1.2 GiB/s.

Then I though I could write a "native" Win32 pv-like program with a few lines of code. So I did exactly that, and ran it from the Windows command prompt:
generator.exe | native_pv.exe > NUL

This time, however, I got a throughput of only ~500 MiB/s

Interesting observation: The "native_pv.exe" process was constantly using 12.5% of the CPU (on a Quadcore with HT that is), whereas the "generator.exe" process was idling at ~5% CPU usage.

Normally "generator.exe" uses 12.5% of the CPU, i.e. the maximum a single-threaded process can use.

But what's really bugging me is that, if I run the following from within the MSYS2 shell, then I suddenly get the full ~1.2 GiB/s again:
./generator.exe | ./native_pv.exe > /dev/null

This means that, apparently, not my "native_pv.exe" program sucks, but the pipe behaves quite differently in the MSYS2 shell than it does in the native Windows command prompt

Any ideas what is going on here?

(I would have expected that, if at all, the MSYS2 shell should be slower than the native command prompt, because it adds another emulation layer, i.e. msys-2.0.dll, on top of the Win32 API)
__________________
There was of course no way of knowing whether you were being watched at any given moment.
How often, or on what system, the Thought Police plugged in on any individual wire was guesswork.



Last edited by LoRd_MuldeR; 6th August 2020 at 17:14.
LoRd_MuldeR is offline   Reply With Quote
Old 5th August 2020, 07:32   #2  |  Link
nevcairiel
Registered Developer
 
Join Date: Mar 2010
Location: Hamburg/Germany
Posts: 10,002
Did you open the pipe in binary mode, to avoid any crazy overhead from Windows processing the stream in text mode?
__________________
LAV Filters - open source ffmpeg based media splitter and decoders
nevcairiel is offline   Reply With Quote
Old 5th August 2020, 15:43   #3  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,146
Quote:
Originally Posted by nevcairiel View Post
Did you open the pipe in binary mode, to avoid any crazy overhead from Windows processing the stream in text mode?
It's actually native Win32 code, so I'm using ReadFile() and WriteFile() directly, thus bypassing the C library (and any text/binary conversion) completely.

Anyway, today I create a little helper application that launches two sub-processes and connects them via pipe. Because, this way, the creation of the pipe is under my control, I can choose the buffer size.

Turns out that this command, in the Windows command prompt, results in ~1.1 GiB/s:
mkpipe.exe generator.exe "|" native_pv.exe > NUL

(Pipe buffer size was set to 1 MiB)
__________________
There was of course no way of knowing whether you were being watched at any given moment.
How often, or on what system, the Thought Police plugged in on any individual wire was guesswork.



Last edited by LoRd_MuldeR; 6th August 2020 at 17:13.
LoRd_MuldeR is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 13:53.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.