Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re: Open3 and bad gut feeling : 357 char STDIN limit?

by coreolyn (Parson)
on Feb 19, 2002 at 15:57 UTC ( #146403=note: print w/ replies, xml ) Need Help??


in reply to Open3 and bad gut feeling

Arg..

I had thought I had overcome the problem of executing large command lines by writing the command line into script that is executed. However, even though the created script runs fine directly from the command line, I have found that if I run it through the attached sub and the line to be executed execeeds 357 characters I 'hang' (MSwin32 - 2000). Additionally I have attempted to implement IO::Handle and IO::Select but apparently not in an effective way. (If it is indeed some type of a buffering problem). I lost all day yesterday to this code and hope someone can pinpoint my ignorance, or am I just exceeding the limitations of open3?

Here's the relevant code. (note: I pushed all the logging to a separate module)

# RETURNS: %OSexec = ( stdout => $stdout, stderr => $stderr pid => $pi +d ) sub OSexecute { my $execute = $_[0]; my $stdin = IO::Select->new(); # not sure what difference <&STDIN and \*STDIN have $stdin->add("\*STDIN"); my $din = IO::Handle->new(); $stdin->add($din); my $stdout = IO::Select->new(); $stdout->add(\*STDOUT); my $dout = IO::Handle->new(); $stdout->add($dout); my $stderr = IO::Select->new(); $stderr->add(\*STDERR); my $derr = IO::Handle->new(); $stderr->add($derr); my ( $pid, $val, $out, $err ); my ( @stdin, @stdout, @stderr ); my $debug = "false"; #$debug = "true"; if ( $debug =~ /true/i ) { print "OSify::Execute::OSexecute::\$execute = $execute\n"; } # Here is the actual execution eval { print "Executing $execute\n"; $pid = open3( $din, $dout, $derr, $execute ); print "Waiting for pidout\n"; # waitpid waits for the proces to exit # $val could be used as a means to determine status # while waiting if that functionality becomes needed. $val = waitpid(-1,0); #wait's for process to complete # Process the results # Standard Out my $line; my @stdout = <$dout>; foreach $line (@stdout) { chomp($line); $out = $out . $line; } if ( ! $out ) { $out = 1; } # Standard Error @stderr = <$derr>; foreach $line ( @stderr ) { chomp($line); $err = $err . $line; } if ( ! $err ) { $err = 1; } }; $@ && die "OSify::OSexecute died upon execution of\n$execute\nWith +$@"; # Question remains what activity qualifies as draining the buffer? $din->flush(); $din->close; $dout->flush(); $dout->close; $derr->flush(); $derr->close; my %OSexec = ( stdout => $out, stderr => $err, pid => $pid, ); print "Execute Finished with @{[%OSexec]}\n"; return %OSexec; }

Here's a very typical script it will execute. The antcall.bat doesn't even have to exist to duplicate the problem.

E:\\cccharv\\JDK\\Release\\scripts\\antcall.bat -logfile E:\\cccharv\\ +testApp\\testVer\\Unit_Test\\antscripts\\build\\logs\\20020219-082532 +_Ant_build_testApp.txt -buildfile E:\\cccharv\\testApp\\testVer\\Unit +_Test\\antscripts\\buildTESTAPP.xml compileTESTAPPClean buildTESTAPPJ +ar buildTESTAPPWar buildClientControllerEJB buildTESTAPPSessionEJB bu +ildTESTAPPEar

Getting rid of the last three characters eliminates the hang

coreolyn


Comment on Re: Open3 and bad gut feeling : 357 char STDIN limit?
Select or Download Code
Re: Re: Open3 and bad gut feeling : 357 char STDIN limit?
by coreolyn (Parson) on Feb 19, 2002 at 18:45 UTC

    Well I've narrowed the problem to STDOUT. If I turn @echo off I've eliminated the problem. Which for this particular app is ok, but I'd sure like to see the problem eliminated completely and to get a better handle on filehandles/buffering.

    Update:Aw frick'n frack'n bull riffin DOS frmbln grrr..

    @echo off just fixed the Perl problem now the OS can't handle it... This is why I quit developing on DOS!!!!

    coreolyn ... mumble mumble mumble.
Re: Re: Open3 and bad gut feeling : Finally
by coreolyn (Parson) on Feb 19, 2002 at 22:20 UTC

    Welp.. turns out that if I pass the command+args as an array instead of a string to Open3 the problem disapears. I don't even have to create an script to run large command lines.

    The DOS problem was eliminated by slurping up the args via set VAR=%*.

    So to put an end to this here's the code as I am going to run with it.

    # RETURNS: %OSexec = ( stdout => $stdout, stderr => $stderr pid => $pi +d ) sub OSexecute { my @execute = @_; my $stdin = IO::Select->new(); # not sure what difference <&STDIN and \*STDIN have $stdin->add("\*STDIN"); my $din = IO::Handle->new(); $din->autoflush(1); $stdin->add($din); my $stdout = IO::Select->new(); $stdout->add(\*STDOUT); my $dout = IO::Handle->new(); $dout->autoflush(1); $stdout->add($dout); my $stderr = IO::Select->new(); $stderr->add(\*STDERR); my $derr = IO::Handle->new(); $derr->autoflush(1); $stderr->add($derr); my ( $pid, $val); my $debug = "false"; #$debug = "true"; if ( $debug =~ /true/i ) { print "OSify::Execute::OSexecute::\$execute = @execute\n"; } # Here is the actual execution eval { $pid = open3( $din, $dout, $derr, @execute ); # waitpid waits for the proces to exit # $val could be used as a means to determine status # while waiting if that functionality becomes needed. $val = waitpid(-1,0); #wait's for process to complete }; $@ && die "OSify::OSexecute died upon execution of\n@execute\nWith +$@"; # Gather the results my $line; # Standard Out my @stdout = <$dout>; my $out; foreach $line (@stdout) { $line = OSify::Utility::trimSpaces($line); $out = $out . $line; } if ( ! $out ) { $out = 1; } # Standard Error my @stderr = <$derr>; my $err; foreach $line ( @stderr ) { $line = OSify::Utility::trimSpaces($line); $err = $err . $line; } if ( ! $err ) { $err = 1; } # Flush and close the Filehandles $din->flush(); $din->close; $dout->flush(); $dout->close; $derr->flush(); $derr->close; my %OSexec = ( stdout => $out, stderr => $err, pid => $pid, ); return %OSexec; }
    coreolyn

      This code can be trimmed down a lot.

    • $stdin, $stdout and $stderr are never used, so they should be deleted.
    • Flushing does nothing for input handles, so $dout->autoflush(1);, $derr->autoflush(1);, $dout->flush(); and $derr->flush(); do nothing.
    • $din->flush(); is useless for two reason: 1) The program on the other side of the pipe has ended, and 2) the close does a flush.
    • $din->autoflush(1); is not needed since we don't write to $din. Even if we did, open3 makes it autoflush for us.
    • Since we don't do anything with $din, $dout and $derr anymore, we can let open3 create them for us.
    • Filehandles are automatically closed when they go out of scope.
    • The $pid is for a dead process, so why return it to the caller?
    • It's more common to use 0/1 for boolean instead of "false"/"true". Deviations make it less readable, and strings make it more complicated. Using a regular expression to check the equality of two strings is overkill. eq is the code for that.
    • Reformatting $out and $err has nothing to do with OSexecute and should be done by the calling function if it so desires.
    • There are many small problems.

    • $stdin->add("\*STDIN"); should be $stdin->add(\*STDIN);.
    • $val = waitpid(-1, 0); should be $val = waitpid($pid, 0);. $val isn't used, so the assignment can be ommited as well.
    • "@execute" will cram all the arguments into one word.
    • Why change a stdout and stderr to 1 if it's empty, when it's just as easy to check for the empty string ($str eq '' and length($str)
    • $stdout is changed from to 1 if it's '0'. Same for $stderr.
    • Returning a hash is... odd. If the code did return ($stdout, $stderr);, the caller could very simply do my ($stdout, $stderr) = OSexecute(...);
    • Cleaned Up Code

      # RETURNS: ( $stdout, $stderr ) Both are refs to an array of lines. sub OSexecute { my @execute = @_; local $, = ' '; # for print "...@execute..." my $debug = 0; #$debug = 1; print("OSexecute(@execute)\n") if ($debug); # Here is the actual execution. my $pid = eval { open3($din, $dout, $derr, @execute) }; die "OSexecute(@execute): $@" if ($@); # Wait for process to complete. waitpid($pid, 0); # Gather the results my @stdout = <$dout>; my @stderr = <$derr>; # We should check the return code of the child. # Gotta trap SIGPIPE for that. return ( \@stdout, \@stderr ); }

      There's a big problem

      { my $file_name; foreach $file_name ('c:\\tinyfile.txt', 'c:\\biggfile.txt') { my ($stdout, $stderr); print("$file_name\n", ("=" x length($file_name))."\n", "\n"); ($stdout, $stderr) = OSexecute('cmd.exe', '/c', 'type', "\"$file_name\""); print("stdout\n", "------\n", @$stdout); print("Nothing was sent to STDOUT.\n") unless (@$stdout); print("\n"); print("stderr\n", "------\n", @$stderr); print("Nothing was sent to STDERR.\n") unless (@$stderr); print("\n", "\n"); } }

      outputs

      c:\tinyfile.txt =============== stdout ------ foo bar bla stderr ------ Nothing was sent to STDERR. c:\biggfile.txt =============== *** HANGS ***

      The problem is that file handles (including $dout and $derr) have a buffer that's limited in size. biggfile.txt is less than 2KB, which is really quite small, so this needs to be fixed.

      Fix

      # RETURNS: ( $stdout, $stderr ) Both are refs to an array of lines. sub OSexecute { my @execute = @_; my @stdout; my @stderr; local $, = ' '; # for print "...@execute..." my $debug = 0; #$debug = 1; print("OSexecute(@execute)\n") if ($debug); # Here is the actual execution. my $pid = eval { open3($din, $dout, $derr, @execute) }; die "OSexecute(@execute): $@" if ($@); my $select = IO::Select->new(); $select->add($dout); $select->add($derr); my @ready; my $fh; # Gather the results while (@ready = $select->can_read()) { foreach $fh (@ready) { push(@stdout, <$fh>) if ($fh == $dout); push(@stderr, <$fh>) if ($fh == $derr); } } # Wait for process to complete and reap it. waitpid($pid, 0); # We should check the return code of the child. # Gotta trap SIGPIPE for that. return ( \@stdout, \@stderr ); }

      Fixed?

      Oops! select() (and IO::Select) doesn't work in Windows. That sucks. It means we're gonna have to use threads!!! I have no experience with threads, so maybe another day.

        Big Sincere Thank You for your comments and thoughts on this node!

        As I mentioned earlier I had no problems with it and have exipirienced no issues other than the 'hang' on NT. It has come up a couple of times in the last couple of years but rerunning the base app has worked on both occasions, and I've always had $stderr returned.

        (As for returning the hash... I created a logging scheme ( Prior to log4 pardigms that worked with the hash as a parameter)

        I will make some time to dig back into this now that I've finally gotten somethings to ponder and consider.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://146403]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (8)
As of 2014-11-27 04:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (179 votes), past polls