Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re: Synchronizing STDERR and STDOUT

by coreolyn (Parson)
on Sep 21, 2006 at 13:56 UTC ( #574129=note: print w/ replies, xml ) Need Help??


in reply to Synchronizing STDERR and STDOUT

While there's much that could be cleaned up, and in spite of my initial gut feeling in Open3 and bad gut feeling, This open 3 module has provided me what you seek in several large enterprise applications without any problems.

Here's the code as it's been running in production without change for the last 4 years

package OSify::Execute; =head1 NAME C<OSify::Execute> -- Provides Access to the system Shell =head1 SYNOPSIS C<B<use OSify::Execute;>> =head1 DESCRIPTION Purpose: Provide a consistent reusable loged common Perl command interface to any supported system shell. Currently only supports solar +is and MSwin32 ( see L<C<OSify::OSify>|OSify> ). =cut use diagnostics; use strict; ###################################################################### +######## # # # This Module is the bottom line in package OSify:: This is where the + # # duct tape directly acceses the OS. Consequently, for purposes of sec +urity # # and extensibility with other Teams || Departments, this code and in + # # particular this module, are heavily annotated. (More notes than code +) # # # # Needs: Currently lacks a security wrapper. This module at times has +an # # effective ID of root. A package OSify::SecurityAPI (LDAP?) should be + # # pursued whenever possible. See Notes with each function as to short + and # # longterm improvements. # # # ###################################################################### +######## # Standard Perl libraries use IPC::Open3; # for OSexecute use IO::Handle '_IOFBF'; # for OSexecute use IO::Handle '_IOLBF'; use IO::Select; # This module currently requires the following functions use Exporter(); our @EXPORT = qw[ ]; our @EXPORTOK = qw[ OSrun ]; =pod =head2 C<B<OSrun()>> Forks a process into the OS via C<OSify::Execute::OSexecute()> It returns the pid, stdout and stderr in a hash, logging the execution to the specified logfile. Intended as the common interface to the execution of the command. This is the only exported function which somewhat forces commands requesting execution to be 'wrapped'. Currently it acts as an optional wrapper to the logger but future functionality ( LDAP? ) should be integrated either here or in a similiar module. Any Calling program must handle the C<STDERR> to determine if it worke +d or not. =over 4 =item usage: C<%result = B<OSexec("OSrun", "logfile");>> C<%result = ( stdout =E<gt> $stdout, stderr =E<gt> $stderr pid =E<gt> +$pid );> =back =cut ###################################################################### +######## # Potential problem. Learning how to either disable or take advantage # of Perl's Filehandle buffering. While the latter would provide # obvious performance benifits. Could run into problems with the # writing to the logfile writing otherwise. ( Refer: Suffering from # Buffering http://perl.plover.com/FAQs/Buffering.html ) sub OSrun { my $execute = $_[0]; my $logfile = $_[1]; my $silent = 0; if ( $_[2] ) { $silent = $_[2]; } # See if execution would be too large for the input # buffer. If so Create and execute a script instead. my @execute; @execute = split /\s/, $execute; # Interface the logger if ( $logfile && $silent ) { if ( $silent > 0 ) { OSify::OSify::OSlogger( $logfile, "Executing $execute", 1 +); } } # Send the command to the executioner my %OSreturn; %OSreturn = OSexecute( @execute ); # Return the output return %OSreturn; } =pod =head2 B<C<OSexecute()>> Forks a process into the OS returning the pid, stdout and stderr in a +hash. Intended to be strictly the minimum required functionality to execute +a command and return all of its relevant information. =over 4 =item usage: C<%result = B<OSexecute("Commandline");>> C<%result = ( stdout =E<gt> $stdout, stderr =E<gt> $stderr pid =E<gt> +$pid );> =back =cut # $@ should NEVER report errors! Termination (die) is written as manda +ntory. sub OSexecute { my @execute = @_; my $stdin = IO::Select->new(); $stdin->add("\*STDIN"); my $din = IO::Handle->new(); $din->autoflush(1); $stdin->add($din); my $stdout = IO::Select->new(); $stdout->add(\*STDOUT); my $dout = IO::Handle->new(); $dout->autoflush(1); $stdout->add($dout); my $stderr = IO::Select->new(); $stderr->add(\*STDERR); my $derr = IO::Handle->new(); $derr->autoflush(1); $stderr->add($derr); OSify::PerlFunc::debug("Executing @execute\n", 10 ); my $pid = 1; # Here is the actual execution my $val = -1; # This eval is looped to allow for retries when intermitant # Resouce temporarily unavailable problems occur. my ($i, $retry, $napTime); $napTime = 5; $retry = 3; for ( $i=1; $i<=$retry; $i++ ) { eval { $pid = open3( $din, $dout, $derr, @execute ); while ( $val != -1 ) { # waitpid waits for the proces to exit # $val could be used as a means to determine status # while waiting if that functionality becomes needed. $val = waitpid(-1,0); #wait's for process to comple +te } }; if ( $@ && $i > $retry ) { die "OSify::OSexecute died upon execution of\n@execute\nWi +th $@"; } elsif ( $@ =~ /Resource temporarily unavailable/i ) { OSify::PerlFunc::debug("Failed to execute\n@execute\non at +tempt number $i\nResource temporarily unavailable.\n Retrying in $nap +Time secs\n", 100); undef($@); sleep($napTime); } elsif ( ! $@ ) { $i = $retry; undef($@); } } # Gather the results my $line; # Standard Out my @stdout = <$dout>; my $out; foreach $line (@stdout) { OSify::PerlFunc::debug( "@execute STDOUT processing \$line = $ +line", 10); $line = OSify::PerlFunc::trimSpaces($line); $out = $out . $line . "\n"; } if ( ! $out ) { $out = 1; } # Standard Error my @stderr = <$derr>; my $err; foreach $line ( @stderr ) { OSify::PerlFunc::debug( "@execute STDERR processing \$line = $ +line", 10); $line = OSify::PerlFunc::trimSpaces($line); $err = $err . $line . "\n"; } if ( ! $err ) { $err = 1; } # Flush and close the Filehandles $din->flush(); $din->close; $dout->flush(); $dout->close; $derr->flush(); $derr->close; undef $stdin; undef $stdout; undef $stderr; my %OSexec = ( stdout => $out, stderr => $err, pid => $pid, ); return %OSexec; } 1; =pod =head1 REQUIRES Standard Perl Modules: L<C<IPC::Open3>|Open3>, L<C<IO::Handle>|Handle>, L<C<IO::Select>|selec +t> OSify Modules: L<C<OSify::PerlFunc>|PerlFunc>, L<C<OSify::OSify>|OSify> =head1 Author =head1 See Also L<C<OSify::OSify>|OSify>, L<C<OSify::FileFunc>|FileFunc>, L<C<OSify::PerlFunc>|PerlFunc>, L<C<OSify::Ultimail>|Ultimail> =head1 =cut =cut

Edited by planetscape - added readmore tags

( keep:0 edit:11 reap:0 )


Comment on Re: Synchronizing STDERR and STDOUT
Download Code
Re^2: Synchronizing STDERR and STDOUT
by OfficeLinebacker (Chaplain) on Sep 21, 2006 at 14:27 UTC
    coreolyn, sweet stuff. I don't fully understand it yet, but the "Suffering from Buffering" article is proving to be a nice read. Does your program keep the stdout and stderr output in such a way that you can reassemble it in the order it actually was generated?

    Terrence

    _________________________________________________________________________________

    I like computer programming because it's like Legos for the mind.

      I use two separate logs for stderr and stdout. To be completely honest I haven't had a situation where I've needed to make sure they are completely in sync. So maybe I was overly smug in my assumption. You should read the comments that are included in the full node of Open3 and bad gut feeling apparently there's is a lot of extraneous ( read useless code ). I never was given an opportunity, nor did the need arise to refactor it.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://574129]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (6)
As of 2014-07-26 18:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (178 votes), past polls