Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked

open3 and IO::Select

by stephens2k (Novice)
on Nov 19, 2007 at 21:20 UTC ( #651768=perlquestion: print w/replies, xml ) Need Help??
stephens2k has asked for the wisdom of the Perl Monks concerning the following question:


I'm trying to write a script that will execute a process in the background, monitoring STDOUT and STDERR to obtain data, then print "QUIT" on STDIN to kill off the application. I'm developing on XP but this will run on *nix.

I'm attempting to use open3 and IO::Select to read both STDOUT and STDERR without blocking. It ain't working...

  • if I dumpValue the select handle I never see STDERR get set.
  • I never got into the while because "$sel->can_read" appears to never return a ready handle
  • I'm open to other approaches here.

    my ($read, $write, $error); my $pid = open3( $write, $read, $error, "$em_command &") || die ERROR. +" : Unable to execute "; write_log (DEBUG, "Maintenance Proxy pid = $pid"); # exception occured if ($pid =~ /^open3:/) { write_log (ERROR, $pid); exit 1; } my $sel = IO::Select->new(); $sel->add($read); dumpValue ($sel); print "---\n"; $sel->add($error); dumpValue ($sel); print "---\n"; my ($rh, $inStr); while(my @ready = $sel->can_read) { foreach my $handle (@ready) { if ($handle == $error) { my $numBytesRead = sysread($rh, $inStr, 1024); print "ERROR: $inStr"; } else { my $numBytesRead = sysread($rh, $inStr, 1024); print "INFO: $inStr"; } $sel->remove($handle) if eof($handle); } }

    Replies are listed 'Best First'.
    Re: open3 and IO::Select
    by ikegami (Pope) on Nov 19, 2007 at 22:21 UTC

      None of the things I suggested earlier would fix the problem you mentioned. After some debugging, I found that open3 defaults to using the same pipe for the child's STDOUT and STDERR. (WTF?!? I'd use open2 if I wanted to do that.)

      One way around that is to vivify the handles yourself. In the tested code below, that's done using

      use Symbol qw( gensym ); my ($to_child, $fr_child, $fr_child_err) = map gensym, 1..3;

      And here's the code I used for testing.

      #!/usr/bin/perl use strict; use warnings; use IO::Select qw( ); use IPC::Open3 qw( open3 ); use Symbol qw( gensym ); my @cmd = ( 'perl', '-e', <<__EOI__ use IO::Handle qw( ); # Usually want autoflush=1 for pipes. STDOUT->autoflush(0); STDERR->autoflush(0); print STDERR ('a') for 1..10000; STDERR->flush(); print STDOUT ('b') for 1..10000; STDOUT->flush(); print STDERR ('c') for 1..10000; STDERR->flush(); print STDOUT ('d') for 1..10000; STDOUT->flush(); __EOI__ ); { my ($to_child, $fr_child, $fr_child_err) = map gensym, 1..3; my $pid = open3($to_child, $fr_child, $fr_child_err, @cmd); my $sel = IO::Select->new(); $sel->add($fr_child); $sel->add($fr_child_err); while (my @ready = $sel->can_read()) { foreach my $handle (@ready) { if ($handle == $fr_child) { my $bytes_read = sysread($handle, my $buf='', 1024); if ($bytes_read == -1) { warn("Error reading from child's STDOUT: $!\n"); $sel->remove($handle); next; } if ($bytes_read == 0) { print("Child's STDOUT closed\n"); $sel->remove($handle); next; } printf("%4d bytes read from child's STDOUT\n", $bytes_read +); } elsif ($handle == $fr_child_err) { my $bytes_read = sysread($handle, my $buf='', 1024); if ($bytes_read == -1) { warn("Error reading from child's STDERR: $!\n"); $sel->remove($handle); next; } if ($bytes_read == 0) { print("Child's STDERR closed\n"); $sel->remove($handle); next; } printf("%4d bytes read from child's STDERR\n", $bytes_read +); } } # For demonstration purposes only. # Should cause some STDOUT and STDERR reads to become interlaced use Time::HiRes qw( sleep ); sleep(0.1); } for (;;) { my $pid = wait(); last if $pid == -1; # Check the error code of the child process if desired. } }
        After some debugging, I found that open3 defaults to using the same pipe for the child's STDOUT and STDERR.
        Yes. It's easy to skip over that fact, but it is in the documentation:
               Extremely similar to open2(), open3() spawns the given $cmd and con-
               nects RDRFH for reading, WTRFH for writing, and ERRFH for errors.  If
               ERRFH is false, or the same file descriptor as RDRFH, then STDOUT and
               STDERR of the child are on the same filehandle.
        So the only filehandle that needs to be directly set is ERRFH. You can leave the other two alone.
        (WTF?!? I'd use open2 if I wanted to do that.)
        Not in the same way. You might use a shell to redirect STDERR to STDOUT and then read that new (combined) stream with open2, but you cannot use open2 to read STDERR directly. That's a different behavior from open3 making both streams available on the same perl filehandle (without needing an intervening shell for the redirection).

          I realise I come very late in this exchange, but just in case people are still looking for answers, I got it to work with cygwin perl on Windows XP with the following tweaks :

          use IO::Select; use IO::Socket; use IO::Handle; use IPC::Open3;
          $Pin = new IO::Handle; $Pin->fdopen(10, "w"); $Pout = new IO::Handle; $Pout->fdopen(11, "r"); $Perr = new IO::Handle; $Perr->fdopen(12, "r"); $Proc = open3($Pin, $Pout, $Perr, $cmdline);
          $select->add($Pin); $select->add($Pout); $select->add($Perr);

          For $cmdline I used bash, to test things by hand (with STDIN added to my select and printing on $Pin what I get on my perl process' stdin).

          It works beautifully.
    Re: open3 and IO::Select
    by ikegami (Pope) on Nov 19, 2007 at 21:25 UTC

      Are you doing your testing in Windows or Unix? select (the underlying system call used by IO::Select) only works for sockets in Windows.

      "$em_command &" is silly, complicates things, and might even cause problems. Simply $em_command is needed.

      open3 throws an exception on error. It doesn't return, much less with a string matching /^open3:/.

      $handle contains the handle, but you pass $rh to sysread.

        I'm trying to test on Windows. It will be installed on Unix when we have an available test server ready.

        OK, I can kill the bckground on the command.

        I'll test these changes tomorrow.


    Re: open3 and IO::Select
    by gloryhack (Deacon) on Nov 20, 2007 at 05:21 UTC
      You might have a look at IPC::Run. It's by far the easiest/fastest way (of which I'm aware, anyway) to implement the functionality you're looking for.

        Have you tried to use this on Win32? Have you seen the list of win32 limitations? In particular:

        IPC::Run uses helper processes, one per redirected file, .... This is a waste of resources and will change in the future to either use threads (instead of helper processes) or a WaitForMultipleObjects call (instead of select).

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
          I haven't tried anything at all on Win32 in the past 10 years, but the OP said he was going to test on Windows and deploy on "*nix" so it might not be such a bad idea.
    Re: open3 and IO::Select
    by salva (Abbot) on Nov 20, 2007 at 10:26 UTC
      You can not do that on Windows using IPC::Open3 because select only works with sockets (and this is a OS limitation, not perl fault).

      The alternative is to use named pipes (see Win32::Pipe) that have an API that allows to do non-blocking IO, but that API is incompatible with any other out there, so you will have to write your programs (both parent and children) specifically for it, and don't expect then to run on Unix/Linux later.

      Another alternative is to open a socket pair as a TCP connection through the loopback interface (newer perl versions already implement socketpair this way). But on my experience, on Windows, not all applications like having their stdin/stdout/stderr redirected to a socket.


        Thanks for the replies. I'm trying to use only modules that are in the default perl distribution because my sysops won't install anything else (thanks, guys). This is the 5.8.2 distribution on my AIX host.

        IPC::Run is not available in the 5.8.2 distribution.

        I've made it work be reading STDIN and STDERR in separate loops. Not elegant at all but it works on Windows. I expect I'll have to make changes when it is installed on AIX.

        Thanks, Bill