IPC::Open3 failure on Win32

Ovid has asked for the wisdom of the Perl Monks concerning the following question:

I'm in quite a serious dilemma with TAPx::Parser. It's getting close to the point of being beta, but there's a nasty Windows bug that I cannot solve because I don't have access to a Windows box and since I have no experience with inter-process communications on Windows, I wouldn't even know where to start.

I have the following code in TAPx::Parser::Source:

use IPC::Open3;
use IO::Select;
use IO::Handle;

#<snip>

my $stdout        = IO::Select->new();
my $stdout_handle = IO::Handle->new();
$stdout_handle->autoflush(1);
$stdout->add( \*STDOUT );
$stdout->add($stdout_handle);

if ( my $pid = open3( undef, $stdout_handle, undef, $command ) ) {
    return TAPx::Parser::Iterator->new($stdout_handle);
}
else {
    $self->exit( $? >> 8 );
    $self->error("Could not execute ($command): $!");
    return;
}
[download]

This code is intended to merge STDIN and STDOUT and keep them synchronized (no buffering problems). It runs fine on Mac OS X, Linux, FreeBSD, OpenBSD, etc. However, Win32 users keep reporting errors similar to:

# Failed test ''bignum' should parse successfully'
# at t/010-regression.t line 2257.
# open3: IO::Pipe: Can't spawn-NOWAIT: No such file or directory at
C:\TAPx-Parser-0.50_03\blib\lib/TAPx/Parser/Source.pm line 103
[download]

(Line 103, not surprisingly, is the IPC::Open3::open3 command.)

I can't debug this since I don't have a Windows box handy. Googling and reading the Perl Cookbook have also not helped. Further, my research on this site shows many people addressing questions about IPC::Open3 shows people posting snippets, but often not fixing the original poster's code. Since I can't test anything, I need to hopefully get this right on the first try :(

The first argument to open3 is undefined since I don't need to write anything to the child. Should I include a handle and simply close it right away? The third argument is undefined because this forces STDERR an STDOUT to merge, which is the desired behavior. I would really like to get TAPx::Parser production ready and this is one of my last major blockers.

As the above code suggests, the $stdout_handle is read in TAPx::Parser::Iterator.

I really hate to be one to say "do my work for me", but I'm really stuck on this one and have no easy way to verify that it works on Windows.

Miscellaneous notes:

I don't know how to capture the exit status with the above code, either. This may not be a big deal, but it would be really nice to solve.

I've tried reading the source of IPC::Cmd, but I can't figure out how he's fixed the Windows problem. I also can't use that as a prerequisite as I need to ensure that no non-core modules are used (there's a desire expressed on the Perl-QA wiki to have a new TAP::Harness in the core and I have grandiose dreams that it's TAPx::Harness).

Update: I forgot to mention that apparently this is apparently only failing for my t/010-regression.t tests. The other tests all appear to be fine (and all use the same open3 command). The difference in the regression tests is that they run a bunch of tests in the t/sample-tests/ directory. So I have a test program, t/010-regression.t, which is run via the open3 command (apparently successfully), which in turn tries to run other tests via the open3 command (unsuccessfully) and generates the error above.

Cheers,
Ovid

New address of my CGI Course.

Comment on IPC::Open3 failure on Win32 Select or Download Code

Replies are listed 'Best First'.
Re: IPC::Open3 failure on Win32 by Corion (Patriarch) on Jan 09, 2007 at 19:17 UTC
What is in `$command` ? Most likely that's the reason why the `open3` call is failing ... Win32 shell invocations are tricky and if calling Perl scripts you'll likely have to make sure to prepend a (properly quoted) `$^X`.	[reply] [d/l] [select]
Re^2: IPC::Open3 failure on Win32 by Ovid (Cardinal) on Jan 09, 2007 at 19:28 UTC
One example would be: `/usr/local/bin/perl t/010-regression.t` I forgot to mention that apparently this is apparently only failing for my t/010-regression.t tests. The other tests all appear to be fine (and all use the same open3 command). The difference in the regression tests is that they run a bunch of tests in the `t/sample-tests/` directory. So I have a test program, `t/010-regression.t`, which is run via the open3 command (apparently successfully), which in turn tries to run other tests via the open3 command (unsuccessfully) and generates the error above. I'll repost this bit in the root node as people sometimes don't notice the replies. Cheers, Ovid New address of my CGI Course.	[reply] [d/l]
Re^3: IPC::Open3 failure on Win32 by Corion (Patriarch) on Jan 09, 2007 at 19:49 UTC
It's unlikely that, on Windows, there will be a `perl` in `/usr/local/bin/perl`. I can reproduce what you find locally with my Perl 5.8.5. I patched your module and made it output the strings and they look like the following: `Spawning >>c:\perl\5826ED~1.5\bin\perl.exe -Ilib t\sample-tests\too_m +any<< at lib/TAPx/Parser/Source.pm line 103.` [download] I don't know who or what is unhappy with the whole being a string instead of being a list, but when I patch it to split the string on whitespace just to use it as a list, it works and I get failures due to whitespace issues in your tests (newlines look different on Win32): `my @command = split / +/, $command; warn sprintf "Spawning >>%s<<", join "", @command; if ( my $pid = open3( undef, $stdout_handle, undef, @command ) ) {` [download] So it looks like you have to rework your API so you pass around lists instead of strings to execute, and you'll have to rework your tests so newlines and trailing whitespace aren't significant anymore. ... Spawning >>c:\perl\5826ED~1.5\bin\perl.exe-Ilib*t\sample-tests\too_ma +ny<< at lib/TAPx/Parser/Source .pm line 104. ok 2316 - too_many 1 We should have a result for 1..3 ok 2317 - ... and is_plan should return a reasonable value ok 2318 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::passed ok 2319 - ... and passed should return a reasonable value ok 2320 - ... and tests_planned should return the correct answer ok 2321 - ... and is_ok should return a reasonable value not ok 2322 - ... and raw should return the correct answer # Failed test '... and raw should return the correct answer' # in t\010-regression.t at line 2312. ' got: '1..3 # expected: '1..3' ok 2323 - too_many 2 We should have a result for ok 2324 - ... and has_skip should return a reasonable value ok 2325 - ... and is_test should return a reasonable value ok 2326 - ... and number should return the correct answer ok 2327 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::actual_passed ok 2328 - ... and actual_passed should return a reasonable value ok 2329 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::passed ok 2330 - ... and passed should return a reasonable value ok 2331 - ... and is_ok should return a reasonable value ok 2332 - ... and description should return the correct answer ok 2333 - ... and is_actual_ok should return a reasonable value ok 2334 - ... and has_todo should return a reasonable value ok 2335 - too_many 3 We should have a result for ok 2336 - ... and has_skip should return a reasonable value ok 2337 - ... and is_test should return a reasonable value ok 2338 - ... and number should return the correct answer ok 2339 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::actual_passed ok 2340 - ... and actual_passed should return a reasonable value ok 2341 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::passed ok 2342 - ... and passed should return a reasonable value ok 2343 - ... and description should return the correct answer ok 2344 - ... and is_ok should return a reasonable value ok 2345 - ... and is_actual_ok should return a reasonable value ok 2346 - ... and has_todo should return a reasonable value ok 2347 - too_many 4 We should have a result for ok 2348 - ... and has_skip should return a reasonable value ok 2349 - ... and is_test should return a reasonable value ok 2350 - ... and number should return the correct answer ok 2351 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::actual_passed ok 2352 - ... and actual_passed should return a reasonable value ok 2353 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::passed ok 2354 - ... and passed should return a reasonable value ok 2355 - ... and description should return the correct answer ok 2356 - ... and is_ok should return a reasonable value ok 2357 - ... and is_actual_ok should return a reasonable value ok 2358 - ... and has_todo should return a reasonable value ok 2359 - too_many 5 We should have a result for ok 2360 - ... and has_skip should return a reasonable value ok 2361 - ... and is_test should return a reasonable value ok 2362 - ... and number should return the correct answer ok 2363 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::actual_passed ok 2364 - ... and actual_passed should return a reasonable value ok 2365 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::passed ok 2366 - ... and passed should return a reasonable value ok 2367 - ... and description should return the correct answer ok 2368 - ... and is_ok should return a reasonable value ok 2369 - ... and is_actual_ok should return a reasonable value ok 2370 - ... and has_todo should return a reasonable value ok 2371 - too_many 6 We should have a result for ok 2372 - ... and has_skip should return a reasonable value ok 2373 - ... and is_test should return a reasonable value ok 2374 - ... and number should return the correct answer ok 2375 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::actual_passed ok 2376 - ... and actual_passed should return a reasonable value ok 2377 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::passed ok 2378 - ... and passed should return a reasonable value ok 2379 - ... and description should return the correct answer ok 2380 - ... and is_ok should return a reasonable value ok 2381 - ... and is_actual_ok should return a reasonable value ok 2382 - ... and has_todo should return a reasonable value ok 2383 - too_many 7 We should have a result for ok 2384 - ... and has_skip should return a reasonable value ok 2385 - ... and is_test should return a reasonable value ok 2386 - ... and number should return the correct answer ok 2387 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::actual_passed ok 2388 - ... and actual_passed should return a reasonable value ok 2389 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::passed ok 2390 - ... and passed should return a reasonable value ok 2391 - ... and description should return the correct answer ok 2392 - ... and is_ok should return a reasonable value ok 2393 - ... and is_actual_ok should return a reasonable value ok 2394 - ... and has_todo should return a reasonable value ok 2395 - too_many 8 We should have a result for ok 2396 - ... and has_skip should return a reasonable value ok 2397 - ... and is_test should return a reasonable value ok 2398 - ... and number should return the correct answer ok 2399 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::actual_passed ok 2400 - ... and actual_passed should return a reasonable value ok 2401 - ... we should get a deprecated warning for TAPx::Parser::Res +ults::Test::passed ok 2402 - ... and passed should return a reasonable value ok 2403 - ... and description should return the correct answer ok 2404 - ... and is_ok should return a reasonable value ok 2405 - ... and is_actual_ok should return a reasonable value ok 2406 - ... and has_todo should return a reasonable value ok 2407 - ... and we should have the correct number of results ok 2408 - 'too_many' should parse successfully ok 2409 - ... we should get a deprecated warning for TAPx::Parser::goo +d_plan ok 2410 - ... and good_plan should return a reasonable value ok 2411 - ... and parse_errors should be the correct amount ok 2412 - ...... and the correct values ok 2413 - ... and tests_run should equal 7 ok 2414 - ... and actual_passed should be the correct amount ok 2415 - ...... and the correct values ok 2416 - ... and todo_passed should be the correct amount ok 2417 - ...... and the correct values ok 2418 - ... and passed should be the correct amount ok 2419 - ...... and the correct values ok 2420 - ... and failed should be the correct amount ok 2421 - ...... and the correct values ok 2422 - ... and skipped should be the correct amount ok 2423 - ...... and the correct values ok 2424 - ... and actual_failed should be the correct amount ok 2425 - ...... and the correct values ok 2426 - ... and wait should return a reasonable value ok 2427 - ... and plan should equal 1..3 ok 2428 - ... and tests_planned should equal 3 ok 2429 - ... and todo should be the correct amount ok 2430 - ...... and the correct values 1..2430 # Looks like you failed 51 tests of 2430. [download]	[reply] [d/l] [select]
Re^4: IPC::Open3 failure on Win32 by Ovid (Cardinal) on Jan 09, 2007 at 19:58 UTC
Re^5: IPC::Open3 failure on Win32 by Corion (Patriarch) on Jan 09, 2007 at 20:12 UTC
Some notes below your chosen depth have not been shown here
Re: IPC::Open3 failure on Win32 by diotalevi (Canon) on Jan 09, 2007 at 20:49 UTC
I don't know how to capture the exit status with the above code, either. This may not be a big deal, but it would be really nice to solve. You're expected to `waitpid( $pid, ... )` to get the exit code or set `$SIG{CHLD} = 'IGNORE'`. You're leaving zombies around if you don't. I've no idea what this means on Windows though. Your code is leaky without that. ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊	[reply] [d/l] [select]
Re: IPC::Open3 failure on Win32 by zentara (Archbishop) on Jan 09, 2007 at 20:03 UTC
Have you seen Perl/Tk App and Interprocess Communication , where it is explained by perlhaq that select won't work on windows reliably on pipes (used by IPC::Open3). I've seen IPC::Run mentioned for use on windows. I'm not really a human, but I play one on earth. Cogito ergo sum a bum	[reply]
Re^2: IPC::Open3 failure on Win32 by Ovid (Cardinal) on Jan 09, 2007 at 20:15 UTC
I haven't seen that link. I'll go read it. However, I know this can be solved since IPC::Cmd has done it. However, I cannot use either that or IPC::Run since those are non-core modules :( Cheers, Ovid New address of my CGI Course.	[reply]
Re: IPC::Open3 failure on Win32 by Steve_p (Priest) on Jan 10, 2007 at 00:26 UTC
Google pointed me to the following thread... Win32, IPC::Open2, IO::Pipe Using a `wait()` would seem to be the answer. Test your modules with bleadperl! rsync -avz rsync://public.activestate.com/perl-current/ . ./Configure -des -Dusedevel -Dprefix=/path/to/test/perl make test make install Now, please test you modules! If you have test failures that don't happen with Perl 5.8.8, send a simplified test case to perlbug at perl.org	[reply] [d/l]
Re: IPC::Open3 failure on Win32 by ikegami (Patriarch) on Jan 10, 2007 at 04:59 UTC
I have found some problems in what you posted. You added something something to the IO::Select before it has been opened. `add` determines the fileno of `$stdout_handl` is undefined and returns without adding anything to `$stdout`. `my $stdout = IO::Select->new(); my $stdout_handle = IO::Handle->new(); $stdout->add($stdout_handle); my $pid = open3( undef, $stdout_handle, undef, $command ); ...error handling...` [download] should be `my $stdout = IO::Select->new(); my $stdout_handle = IO::Handle->new(); my $pid = open3( undef, $stdout_handle, undef, $command ); ...error handling... $stdout->add($stdout_handle);` [download] `$stdout_handle->autoflush(1);` is useless, since you don't write to `$stdout_handle`. I think This code is intended to merge STDIN and STDOUT and keep them synchronized (no buffering problems). should read This code is intended to merge STDOUT and STDERR and keep them synchronized (no buffering problems). but that's still wrong. Merging them does not avoid buffering problems, as you can see from the following snippet. `>perl -e "print qq{abc\n}; warn qq{def\n};" 2>&1 \| perl -pe "s/^/$. /" 1 def 2 abc` [download] Only the child can decide whether it will buffer it's output or not. This cannot be controlled by the parent (unless a particular child provides a configuration option which controls buffering). And of course, thre's the zombie problem diotalevi already mentioned.	[reply] [d/l] [select]
Re: IPC::Open3 failure on Win32 by talexb (Chancellor) on Jan 10, 2007 at 04:17 UTC
It seems you've already got a solution, but I wanted to chime in with my praise for IPC::Run -- a little funky to get going, but once you figure out the quirks, it works absolutely terrifically. Alex / talexb / Toronto "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds	[reply]
Re: IPC::Open3 failure on Win32 by El Linko (Beadle) on Jan 14, 2007 at 23:06 UTC
Looks like the new version (TAPx::Parser 0.50_04) still fails some tests. Many (but not all) of the failed tests look like they are caused by windows line endings. Read more... (64 kB)	[reply]


Your skill will accomplish what the force of many cannot
	PerlMonks