|
BazB has asked for the wisdom of the Perl Monks concerning the following question:
I'm attempting to write a wrapper for some external programs that I'll be calling from a Perl script.
I need to capture all the inputs and outputs - STDERR being the most important.
After looking through The Camel and around the Monastery, I found IPC::Open3.
I've already looked at the following nodes, but I'm still unable to get a script that performs as I expect:
For testing purposes I'm just using a simple Perl script as the command to be run by open3(), rather than using the binary. I've also tried using cat as well.
This script is as follows:
#!/usr/local/bin/perl
# Test script to write to STDOUT
# and STDERR.
use warnings;
use strict;
print STDOUT "StdOut!\n";
print STDERR "StdErr!\n";
The main script (using IPC::Open3) is:
#!/usr/local/bin/perl
# Script to test IPC::Open3
# Runs, displays $pid to STDOUT, but opened
# files are empty.
use strict;
use warnings;
use diagnostics;
use IPC::Open3;
$|++;
my $cmd = "/home/baz/test.pl";
open(ERRLOG, ">error.log") or die "Can't open error log! $!";
open(OUTPUT, ">output.log") or die "Can't open output log! $!";
my $pid = open3(\*STDIN, \*OUTPUT, \*ERRLOG, $cmd) or die "$!";
print "PID was $pid\n";
close(ERRLOG) or die "Can't close filehandle! $!";
close(OUTPUT) or die "Can't close filehandle! $!";
I'm expecting this script to write the output from my test.pl script to output.log and errors from the test script to error.log, however both files are created, but empty.
Have I got the wrong end of the stick somewhere?
|
Re: IPC::Open3 woes
by abstracts (Hermit) on Mar 10, 2002 at 22:22 UTC
|
Hello,
Here is a script that does what you wanted to to with enough comments to explain every step:
#!/usr/bin/perl -w
# Script to test IPC::Open3
# Runs, displays $pid to STDOUT, but opened
# files are empty.
use strict;
use warnings;
use diagnostics;
use IPC::Open3;
use IO::Select; # for select
use Symbol; # for gensym
$|++;
my $cmd = "./test.pl";
open(ERRLOG, ">error.log") or die "Can't open error log! $!";
open(OUTPUT, ">output.log") or die "Can't open output log! $!";
my ($infh,$outfh,$errfh); # these are the FHs for our child
$errfh = gensym(); # we create a symbol for the errfh
# because open3 will not do that for us
my $pid;
eval{
$pid = open3($infh, $outfh, $errfh, $cmd);
};
die "open3: $@\n" if $@;
print "PID was $pid\n";
my $sel = new IO::Select; # create a select object to notify
# us on reads on our FHs
$sel->add($outfh,$errfh); # add the FHs we're interested in
while(my @ready = $sel->can_read) { # read ready
foreach my $fh (@ready) {
my $line = <$fh>; # read one line from this fh
if(not defined $line){ # EOF on this FH
$sel->remove($fh); # remove it from the list
next; # and go handle the next FH
}
if($fh == $outfh) { # if we read from the outfh
print OUTPUT $line; # print it to OUTFH
} elsif($fh == $errfh) {# do the same for errfh
print ERRLOG $line;
} else { # we read from something else?!?!
die "Shouldn't be here\n";
}
}
}
close(ERRLOG) or die "Can't close filehandle! $!";
close(OUTPUT) or die "Can't close filehandle! $!";
So, basically, open3 gives you the file handles that you can use to read from or write to the child process. It will not automatically pipe the output to a file as you thought. What you need to do is read from the file handles and print them to the files yourself.
Hope this helps,,,
Aziz,,, | [reply]
[d/l] |
|
|
looks like many people beat me to it on this thread. i guess i should expect that now that my days are filled with ksh and not perl {grin}
abstracts this is good code. very good.
but not perfect.
from select (but not listed in IO::Select -- it should be!):
WARNING: One should not attempt to mix buffered I/O (like read() or <FH>) with select(), except as permitted by POSIX, and even then only on POSIX systems. You have to use sysread() instead.
it is a simple task to modify your code to use sysread, and i don't have the time to do it myself now. if nobody's attacked this by wednesday evening, i'll try to post it then.
otherwise, you'll be Suffering from Buffering
~Particle ;Þ
| [reply] |
|
|
This would be essentially the sort of parent-does-the-reading
script I suggested in my longer response below. Nicely done,
to, at least from a cursory reading-over. ++, Aziz... :-)
--rjray
| [reply] |
Re: IPC::Open3 woes
by rjray (Chaplain) on Mar 10, 2002 at 22:36 UTC
|
It's been a while since I used IPC::Open3, though
I have used it in the past. If I am remembering correctly
(the manual page is a little spare, unfortunately), the
filehandles you pass in are then created for you as dupes of
the relevant file handles from the child process. What this
means in the short form is this: opening the two ahead of time
and passing them in did nothing, they were closed and re-opened
using dup() on the STDERR and STDOUT of the child.
More specifically, the idea of open3() is to
give you (your program) the ability to read off those
two file descriptors locally. So, after a successful return
from open3, you should have been able to read off of
OUTPUT and ERRLOG as if you had opened
them for reading. Because of potential blocking issues, you
are encouraged to use select
before trying to read off of either handle.
In your case, where you want the output to go to specific
files, according to the manual page you should open those
filehandles with ">&", not just ">" as you are doing in your
code. Try changing those lines, and see if that helps. If
not, you can try doing the reading yourself.
And don't feel too bad-- I find the manual page here to be
pretty unclear, as well.
--rjray
| [reply] |
Re: IPC::Open3 woes
by Juerd (Abbot) on Mar 10, 2002 at 21:57 UTC
|
I'm expecting this script to write the output from my test.pl script to output.log and errors from the test script to error.log, however both files are created, but empty.
Unfortunately, you can't just link one thing to another using a single filehandle. You'll have to create some data pipe yourself: read out the programs stdout and stderr, and write it back to the files you want. Don't forget to waitpid, after which $? will have the status code.
44696420796F7520732F2F2F65206F
7220756E7061636B3F202F6D736720
6D6521203A29202D2D204A75657264
| [reply] |
|
|
This doesn't quite address the writer's original problem,
though. He isn't expecting that, he's using the (correct)
library module to set up the pipes on his behalf, behind the
scenes. I think a large part of the problem stems from a
lack of clarity in the manual page for IPC::Open3,
at least in this case.
--rjray
| [reply] |
(tye)Re: IPC::Open3 woes
by tye (Sage) on Mar 11, 2002 at 16:02 UTC
|
#!/usr/local/bin/perl
# Script to test IPC::Open3
# Runs, displays $pid to STDOUT, but opened
# files are empty.
use strict;
use warnings;
if( @ARGV ) {
print STDOUT "StdOut!\n";
print STDERR "StdErr!\n";
exit( 0 );
}
#use diagnostics;
use IPC::Open3;
$|++;
my @cmd = ($^X,$0,"Test");
open(ERRLOG, ">error.log") or die "Can't open error log! $!";
open(OUTPUT, ">output.log") or die "Can't open output log! $!";
my $pid = open3("<&STDIN", ">&OUTPUT", ">&ERRLOG", @cmd) or die "$!";
print "PID was $pid\n";
close(ERRLOG) or die "Can't close filehandle! $!";
close(OUTPUT) or die "Can't close filehandle! $!";
defined( my $x= <STDIN> ) or warn "Can't read STDIN: $!\n";
and then says:
PID was 416
readline() on closed filehandle main::STDIN at open3.pl line 32.
Can't read STDIN: Bad file descriptor
So you might want to dup STDIN first:
open( IN, "<&STDIN" ) or die "...: $!\n";
and pass in "<&IN" instead of "<&STDIN".
-
tye
(but my friends call me "Tye") | [reply]
[d/l]
[select] |
Re: IPC::Open3 woes - thanks.
by BazB (Priest) on Mar 11, 2002 at 00:04 UTC
|
Juerd, rjray and particularly abstracts++ !
Yeap, I must say, perldoc IPC::Open3 isn't the greatest, and I was struggling from the word go anyway :)
The code works nicely, although I'll need to spend some time understanding exactly how it works it's magic.
It's just gone midnight here, so that's for tomorrow.
Once again, cheers!
| [reply] |
Re: IPC::Open3 woes
by abstracts (Hermit) on Mar 15, 2002 at 01:59 UTC
|
Hello all,
BazB asked me to post a complete example of getting IPC::Open3 to work with IO::Select. Please comment on the code if you see any problems with it.
Thanks a million.
Aziz,,,
############################################################
#!/usr/bin/perl -w
# this script printf to stdout and stderr. It prints random
# characters and does not flush the output of stdout. stderr
# is autoflushed by default.
# uncomment the line about autoflush STDOUT to see how that
# changes the behavior. Also, you can uncomment the sleep
# line to watch the script in slow motion.
use warnings;
use strict;
use IO::Handle;
#autoflush STDOUT 1;
for (1..100){
my $str = '';
for(1..80){
$str .= ('A'..'Z','a'..'z',0..9)[rand 62];
}
if(int rand 2){ # 50:50 chance
print STDOUT "StdOut:$str";
} else {
print STDERR "StdErr:$str";
}
# sleep 1;
}
############################################################
#!/usr/bin/perl -w
# this script runs the previous script and catches both
# child's stdout and stderr. It prints progress information
# as it goes. It also prints the data it gets to the
# corresponding file.
use strict;
use warnings;
#use diagnostics;
use IPC::Open3;
use IO::Select;
use Symbol;
my $cmd = "./test.pl";
open(ERRLOG, ">error.log") or die "Can't open error log! $!";
open(OUTPUT, ">output.log") or die "Can't open output log! $!";
my ($infh,$outfh,$errfh);
$errfh = gensym(); # if you uncomment this line, $errfh will
# never be initialized for you and you
# will get a warning in the next print
# line.
my $pid;
eval{
$pid = open3($infh, $outfh, $errfh, $cmd);
};
die $@ if $@;
print "IN: $infh OUT: $outfh ERR: $errfh\n";
print "PID was $pid\n";
# now our child is running, happily printing to
# its stdout and stderr (our $outfh and $errfh).
my $sel = new IO::Select; # create a select object
$sel->add($outfh,$errfh); # and add the fhs
# $sel->can_read will block until there is data available
# on one or more fhs
while(my @ready = $sel->can_read) {
# now we have a list of all fhs that we can read from
foreach my $fh (@ready) { # loop through them
my $line;
# read up to 4096 bytes from this fh.
# if there is less than 4096 bytes, we'll only get
# those available bytes and won't block. If there
# is more than 4096 bytes, we'll only read 4096 and
# wait for the next iteration through the loop to
# read the rest.
my $len = sysread $fh, $line, 4096;
if(not defined $len){
# There was an error reading
die "Error from child: $!\n";
} elsif ($len == 0){
# Finished reading from this FH because we read
# 0 bytes. Remove this handle from $sel.
# we will exit the loop once we remove all file
# handles ($outfh and $errfh).
$sel->remove($fh);
next;
} else { # we read data alright
print "Read $len bytes from $fh\n";
if($fh == $outfh) {
print OUTPUT $line;
} elsif($fh == $errfh) {
print ERRLOG $line;
} else {
die "Shouldn't be here\n";
}
}
}
}
# now that the child closed both its handles, I assume it
# exited.
# ps will show you the <defunct> child.
print `ps`;
# go ahead and reap it
waitpid $pid, 0; # wait for it to die
# not it's no more
print `ps`;
close(ERRLOG) or die "Can't close filehandle! $!";
close(OUTPUT) or die "Can't close filehandle! $!";
| [reply]
[d/l] |
|
|
I am hopeing that somebody can add some clarity to this old thread. Can somebody tell me why the following coded hangs if the line which defines the @outlines array is commented out?
I have used open3 in other places w/o reading the the child's output and the process didn't hang. I expect it is because in that case the command didn't have output.
I understand how some of the other example programs in this thread prevent locking while read from open3's handles but it seems strange that this would lock when I am not reading from them and just calling waitpid as soon as possible.
Also, should I explicitly close $wtr, $rtr, and $err?
...
my($wtr,$rdr);
my $err = gensym;
### $logcmd is a command that returns approx. 460 lines.
my $pid = open3($wtr, $rdr, $err, "$logcmd");
my @outlines = <$rdr>; ### if this line is commented out we hang on w
+aitpid
waitpid( $pid, 0 );
...
| [reply]
[d/l] |
|
|
use strict;
use warnings;
use IPC::Open3 qw( open3 );
for (my $size = 1024; ; $size += 1024) {
print("$size\n");
my $pid = open3(
local our $to_chld,
local our $fr_chld,
local our $fr_chld_err,
perl => (
-e => 'print "x" x $ARGV[0]',
$size
)
);
waitpid($pid, 0);
}
...
62464
63488
64512
65536
66560
[hangs]^C
The solution is to redirect the output to /dev/null
use strict;
use warnings;
use IPC::Open3 qw( open3 );
open(local *DEVNULL, '>', '/dev/null') or die;
for (my $size = 1024; ; $size += 1024) {
print("$size\n");
my $pid = open3(
local our $to_chld,
'>&DEVNULL',
'>&DEVNULL',
perl => (
-e => 'print "x" x $ARGV[0]',
$size
)
);
waitpid($pid, 0);
}
...
365568
366592
367616
368640
369664
370688
371712
372736
373760
374784
...
| [reply]
[d/l]
[select] |
|
|
Please somebody do comment, this is great information.
I fear it is now out of date. I can't get this to work under Perl 5.8 any more.
The documentation does not help.
The code hangs on the open3 call.
| [reply] |
|
|
Hi,
I've been trying your example, and it works great on linux (perl 5.6.1), but doesn't work on win2k (activestate perl 5.8.0).
I've managed to track it down as far as the line where you
call can_read, which for some odd reason appears to cause the
loop to exit immediately (probably returning undef).
Calling count() immediately before the can_read shows that
there are still 2 file handles.
I'm a bit stuck on this one, so if anyone has any ideas,
they'd be appreciated.
Thanks.
Miah.
| [reply] |
|
|
Hi,
I am new to open3 and perldoc is too vague to understand
I am trying a simple unix command to run in open3.
open3($stdin, $stdout, $stderr, "gcp -vpfR $xrc $dst")
What i need is -- I want to caputre STDOUT until gcp finishes or there is something on STDERR (on disk full case etc).
while (<$stdout>) {
print "$_";
}
@stderr = <$stderr>;
if (@stderr) { dir "ERROR: gcp\n" }
BUT this is not working -- any suggestions
| [reply] |
|
|
