Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

IPC::Open3 woes

by BazB (Priest)
on Mar 10, 2002 at 19:18 UTC ( #150748=perlquestion: print w/ replies, xml ) Need Help??
BazB has asked for the wisdom of the Perl Monks concerning the following question:

I'm attempting to write a wrapper for some external programs that I'll be calling from a Perl script.

I need to capture all the inputs and outputs - STDERR being the most important.

After looking through The Camel and around the Monastery, I found IPC::Open3.

I've already looked at the following nodes, but I'm still unable to get a script that performs as I expect:

For testing purposes I'm just using a simple Perl script as the command to be run by open3(), rather than using the binary. I've also tried using cat as well.
This script is as follows:

#!/usr/local/bin/perl # Test script to write to STDOUT # and STDERR. use warnings; use strict; print STDOUT "StdOut!\n"; print STDERR "StdErr!\n";

The main script (using IPC::Open3) is:

#!/usr/local/bin/perl # Script to test IPC::Open3 # Runs, displays $pid to STDOUT, but opened # files are empty. use strict; use warnings; use diagnostics; use IPC::Open3; $|++; my $cmd = "/home/baz/test.pl"; open(ERRLOG, ">error.log") or die "Can't open error log! $!"; open(OUTPUT, ">output.log") or die "Can't open output log! $!"; my $pid = open3(\*STDIN, \*OUTPUT, \*ERRLOG, $cmd) or die "$!"; print "PID was $pid\n"; close(ERRLOG) or die "Can't close filehandle! $!"; close(OUTPUT) or die "Can't close filehandle! $!";
I'm expecting this script to write the output from my test.pl script to output.log and errors from the test script to error.log, however both files are created, but empty.

Have I got the wrong end of the stick somewhere?

Comment on IPC::Open3 woes
Select or Download Code
Re: IPC::Open3 woes
by Juerd (Abbot) on Mar 10, 2002 at 21:57 UTC

    I'm expecting this script to write the output from my test.pl script to output.log and errors from the test script to error.log, however both files are created, but empty.

    Unfortunately, you can't just link one thing to another using a single filehandle. You'll have to create some data pipe yourself: read out the programs stdout and stderr, and write it back to the files you want. Don't forget to waitpid, after which $? will have the status code.

    44696420796F7520732F2F2F65206F
    7220756E7061636B3F202F6D736720
    6D6521203A29202D2D204A75657264
    

      This doesn't quite address the writer's original problem, though. He isn't expecting that, he's using the (correct) library module to set up the pipes on his behalf, behind the scenes. I think a large part of the problem stems from a lack of clarity in the manual page for IPC::Open3, at least in this case.

      --rjray

Re: IPC::Open3 woes
by abstracts (Hermit) on Mar 10, 2002 at 22:22 UTC
    Hello,
    Here is a script that does what you wanted to to with enough comments to explain every step:
    #!/usr/bin/perl -w # Script to test IPC::Open3 # Runs, displays $pid to STDOUT, but opened # files are empty. use strict; use warnings; use diagnostics; use IPC::Open3; use IO::Select; # for select use Symbol; # for gensym $|++; my $cmd = "./test.pl"; open(ERRLOG, ">error.log") or die "Can't open error log! $!"; open(OUTPUT, ">output.log") or die "Can't open output log! $!"; my ($infh,$outfh,$errfh); # these are the FHs for our child $errfh = gensym(); # we create a symbol for the errfh # because open3 will not do that for us my $pid; eval{ $pid = open3($infh, $outfh, $errfh, $cmd); }; die "open3: $@\n" if $@; print "PID was $pid\n"; my $sel = new IO::Select; # create a select object to notify # us on reads on our FHs $sel->add($outfh,$errfh); # add the FHs we're interested in while(my @ready = $sel->can_read) { # read ready foreach my $fh (@ready) { my $line = <$fh>; # read one line from this fh if(not defined $line){ # EOF on this FH $sel->remove($fh); # remove it from the list next; # and go handle the next FH } if($fh == $outfh) { # if we read from the outfh print OUTPUT $line; # print it to OUTFH } elsif($fh == $errfh) {# do the same for errfh print ERRLOG $line; } else { # we read from something else?!?! die "Shouldn't be here\n"; } } } close(ERRLOG) or die "Can't close filehandle! $!"; close(OUTPUT) or die "Can't close filehandle! $!";
    So, basically, open3 gives you the file handles that you can use to read from or write to the child process. It will not automatically pipe the output to a file as you thought. What you need to do is read from the file handles and print them to the files yourself.

    Hope this helps,,,

    Aziz,,,

      This would be essentially the sort of parent-does-the-reading script I suggested in my longer response below. Nicely done, to, at least from a cursory reading-over. ++, Aziz... :-)

      --rjray

      looks like many people beat me to it on this thread. i guess i should expect that now that my days are filled with ksh and not perl {grin}

      abstracts this is good code. very good.

      but not perfect.

      from select (but not listed in IO::Select -- it should be!):

      WARNING: One should not attempt to mix buffered I/O (like read() or <FH>) with select(), except as permitted by POSIX, and even then only on POSIX systems. You have to use sysread() instead.

      it is a simple task to modify your code to use sysread, and i don't have the time to do it myself now. if nobody's attacked this by wednesday evening, i'll try to post it then.

      otherwise, you'll be Suffering from Buffering

      ~Particle ;

Re: IPC::Open3 woes
by rjray (Chaplain) on Mar 10, 2002 at 22:36 UTC

    It's been a while since I used IPC::Open3, though I have used it in the past. If I am remembering correctly (the manual page is a little spare, unfortunately), the filehandles you pass in are then created for you as dupes of the relevant file handles from the child process. What this means in the short form is this: opening the two ahead of time and passing them in did nothing, they were closed and re-opened using dup() on the STDERR and STDOUT of the child.

    More specifically, the idea of open3() is to give you (your program) the ability to read off those two file descriptors locally. So, after a successful return from open3, you should have been able to read off of OUTPUT and ERRLOG as if you had opened them for reading. Because of potential blocking issues, you are encouraged to use select before trying to read off of either handle.

    In your case, where you want the output to go to specific files, according to the manual page you should open those filehandles with ">&", not just ">" as you are doing in your code. Try changing those lines, and see if that helps. If not, you can try doing the reading yourself.

    And don't feel too bad-- I find the manual page here to be pretty unclear, as well.

    --rjray

Re: IPC::Open3 woes - thanks.
by BazB (Priest) on Mar 11, 2002 at 00:04 UTC

    Juerd, rjray and particularly abstracts++ !

    Yeap, I must say, perldoc IPC::Open3 isn't the greatest, and I was struggling from the word go anyway :)

    The code works nicely, although I'll need to spend some time understanding exactly how it works it's magic. It's just gone midnight here, so that's for tomorrow.

    Once again, cheers!

(tye)Re: IPC::Open3 woes
by tye (Cardinal) on Mar 11, 2002 at 16:02 UTC

    From "perldoc IPC::Open3" (the second paragraph):

    If WTRFH begins with `<&', then WTRFH will be closed in the parent, and the child will read from it directly. If RDRFH or ERRFH begins with `>&', then the child will send output directly to that filehandle. In both cases, there will be a dup(2) instead of a pipe(2) made.
    So you should be able to do what you want simply by change one line to: my $pid = open3("<&STDIN", ">&OUTPUT", ">&ERRLOG", $cmd) or die "$!";
    Note that this will close those file handles in the parent as can be seen by this test program which does put the desired output into the specified files:
    #!/usr/local/bin/perl # Script to test IPC::Open3 # Runs, displays $pid to STDOUT, but opened # files are empty. use strict; use warnings; if( @ARGV ) { print STDOUT "StdOut!\n"; print STDERR "StdErr!\n"; exit( 0 ); } #use diagnostics; use IPC::Open3; $|++; my @cmd = ($^X,$0,"Test"); open(ERRLOG, ">error.log") or die "Can't open error log! $!"; open(OUTPUT, ">output.log") or die "Can't open output log! $!"; my $pid = open3("<&STDIN", ">&OUTPUT", ">&ERRLOG", @cmd) or die "$!"; print "PID was $pid\n"; close(ERRLOG) or die "Can't close filehandle! $!"; close(OUTPUT) or die "Can't close filehandle! $!"; defined( my $x= <STDIN> ) or warn "Can't read STDIN: $!\n";
    and then says:
    PID was 416 readline() on closed filehandle main::STDIN at open3.pl line 32. Can't read STDIN: Bad file descriptor
    So you might want to dup STDIN first: open( IN, "<&STDIN" ) or die "...: $!\n";
    and pass in "<&IN" instead of "<&STDIN".

            - tye (but my friends call me "Tye")
Re: IPC::Open3 woes
by abstracts (Hermit) on Mar 15, 2002 at 01:59 UTC
    Hello all,

    BazB asked me to post a complete example of getting IPC::Open3 to work with IO::Select. Please comment on the code if you see any problems with it.

    Thanks a million.

    Aziz,,,

    ############################################################ #!/usr/bin/perl -w # this script printf to stdout and stderr. It prints random # characters and does not flush the output of stdout. stderr # is autoflushed by default. # uncomment the line about autoflush STDOUT to see how that # changes the behavior. Also, you can uncomment the sleep # line to watch the script in slow motion. use warnings; use strict; use IO::Handle; #autoflush STDOUT 1; for (1..100){ my $str = ''; for(1..80){ $str .= ('A'..'Z','a'..'z',0..9)[rand 62]; } if(int rand 2){ # 50:50 chance print STDOUT "StdOut:$str"; } else { print STDERR "StdErr:$str"; } # sleep 1; } ############################################################ #!/usr/bin/perl -w # this script runs the previous script and catches both # child's stdout and stderr. It prints progress information # as it goes. It also prints the data it gets to the # corresponding file. use strict; use warnings; #use diagnostics; use IPC::Open3; use IO::Select; use Symbol; my $cmd = "./test.pl"; open(ERRLOG, ">error.log") or die "Can't open error log! $!"; open(OUTPUT, ">output.log") or die "Can't open output log! $!"; my ($infh,$outfh,$errfh); $errfh = gensym(); # if you uncomment this line, $errfh will # never be initialized for you and you # will get a warning in the next print # line. my $pid; eval{ $pid = open3($infh, $outfh, $errfh, $cmd); }; die $@ if $@; print "IN: $infh OUT: $outfh ERR: $errfh\n"; print "PID was $pid\n"; # now our child is running, happily printing to # its stdout and stderr (our $outfh and $errfh). my $sel = new IO::Select; # create a select object $sel->add($outfh,$errfh); # and add the fhs # $sel->can_read will block until there is data available # on one or more fhs while(my @ready = $sel->can_read) { # now we have a list of all fhs that we can read from foreach my $fh (@ready) { # loop through them my $line; # read up to 4096 bytes from this fh. # if there is less than 4096 bytes, we'll only get # those available bytes and won't block. If there # is more than 4096 bytes, we'll only read 4096 and # wait for the next iteration through the loop to # read the rest. my $len = sysread $fh, $line, 4096; if(not defined $len){ # There was an error reading die "Error from child: $!\n"; } elsif ($len == 0){ # Finished reading from this FH because we read # 0 bytes. Remove this handle from $sel. # we will exit the loop once we remove all file # handles ($outfh and $errfh). $sel->remove($fh); next; } else { # we read data alright print "Read $len bytes from $fh\n"; if($fh == $outfh) { print OUTPUT $line; } elsif($fh == $errfh) { print ERRLOG $line; } else { die "Shouldn't be here\n"; } } } } # now that the child closed both its handles, I assume it # exited. # ps will show you the <defunct> child. print `ps`; # go ahead and reap it waitpid $pid, 0; # wait for it to die # not it's no more print `ps`; close(ERRLOG) or die "Can't close filehandle! $!"; close(OUTPUT) or die "Can't close filehandle! $!";
      Hi,

      I've been trying your example, and it works great on linux (perl 5.6.1), but doesn't work on win2k (activestate perl 5.8.0). I've managed to track it down as far as the line where you call can_read, which for some odd reason appears to cause the loop to exit immediately (probably returning undef). Calling count() immediately before the can_read shows that there are still 2 file handles.

      I'm a bit stuck on this one, so if anyone has any ideas, they'd be appreciated.

      Thanks.

      Miah.

        Hi, I am new to open3 and perldoc is too vague to understand
        I am trying a simple unix command to run in open3.
        open3($stdin, $stdout, $stderr, "gcp -vpfR $xrc $dst")


        What i need is -- I want to caputre STDOUT until gcp finishes or there is something on STDERR (on disk full case etc).

        while (<$stdout>) {
        print "$_";
        }
        @stderr = <$stderr>;
        if (@stderr) { dir "ERROR: gcp\n" }


        BUT this is not working -- any suggestions

      I am hopeing that somebody can add some clarity to this old thread. Can somebody tell me why the following coded hangs if the line which defines the @outlines array is commented out?

      I have used open3 in other places w/o reading the the child's output and the process didn't hang. I expect it is because in that case the command didn't have output.

      I understand how some of the other example programs in this thread prevent locking while read from open3's handles but it seems strange that this would lock when I am not reading from them and just calling waitpid as soon as possible.

      Also, should I explicitly close $wtr, $rtr, and $err?

      ... my($wtr,$rdr); my $err = gensym; ### $logcmd is a command that returns approx. 460 lines. my $pid = open3($wtr, $rdr, $err, "$logcmd"); my @outlines = <$rdr>; ### if this line is commented out we hang on w +aitpid waitpid( $pid, 0 ); ...
        Please somebody do comment, this is great information. I fear it is now out of date. I can't get this to work under Perl 5.8 any more. The documentation does not help. The code hangs on the open3 call.

        I am hopeing that somebody can add some clarity to this old thread. Can somebody tell me why the following coded hangs if the line which defines the @outlines array is commented out?

        You have a deadlock.

        • waitpid never returns because the child never exits.
        • The child never exits because it's blocked trying to write a full pipe.
        • The pipe is full because noone is emptying it.

        If it doesn't happen for all your children, it's because they don't output enough data to fill the pipe.

        use strict; use warnings; use IPC::Open3 qw( open3 ); for (my $size = 1024; ; $size += 1024) { print("$size\n"); my $pid = open3( local our $to_chld, local our $fr_chld, local our $fr_chld_err, perl => ( -e => 'print "x" x $ARGV[0]', $size ) ); waitpid($pid, 0); }
        ... 62464 63488 64512 65536 66560 [hangs]^C

        The solution is to redirect the output to /dev/null

        use strict; use warnings; use IPC::Open3 qw( open3 ); open(local *DEVNULL, '>', '/dev/null') or die; for (my $size = 1024; ; $size += 1024) { print("$size\n"); my $pid = open3( local our $to_chld, '>&DEVNULL', '>&DEVNULL', perl => ( -e => 'print "x" x $ARGV[0]', $size ) ); waitpid($pid, 0); }
        ... 365568 366592 367616 368640 369664 370688 371712 372736 373760 374784 ...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://150748]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (7)
As of 2014-09-21 15:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (172 votes), past polls