Re: Redirecting stdout/stderr to pipe

[Thelonius]

I think your main problems is that select does not interact well with buffered I/O, which you use with <READERR> and <READOUT>. You should really use sysread and do a split /\n/ at the very end.

Here's how I might do it, using IPC::Open3 and IO::Select. You should add the waitpid/close from your example:

#!perl -w
use IPC::Open3;
use IO::Select;
use Symbol qw(gensym);
use strict;
my $cmd = shift;
my $g_timeout = shift || 5;
my $g_maxlines = shift || 10;
my @output;
my @err;
$| = 1;
runcmd(\$cmd, \@output, \@err);
print "output = \n";
print "  $_\n" for @output;
print "error = \n";
print "  $_\n" for @err;

sub runcmd {
  my ($cmdref, $outref, $errref) = @_;
  my ($childin, $childout, $childerr);
  $childerr = gensym;

  my $pid = open3($childin, $childout,  $childerr,  $cmd)
    or die "Cannot run cmd '$cmd': $!\n";

  my $select = IO::Select->new or die "Cannot create select object: $!
+\n";
  my @hold_output;
  for (($childout, $childerr)) {
    $select->add($_);
    $hold_output[fileno($_)] = [0, 0, ""]; # eof, lines, buffer;
  }

  $@ = undef;
  eval {
    local $SIG{ALRM} = sub { die "alarm\n" };
    my $deadline = $g_timeout + time;
    my $g_stop = 0;
    alarm($g_timeout + 1);
    while (!$g_stop && $select->count > 0) {
      $! = 0;
      my @ready = $select->can_read($deadline - time);
      if (!@ready) {
        $g_stop = 1;
      }
      for my $handle (@ready) {
        my $fno = fileno($handle);
        my $line;
        my $bytesread = sysread $handle, $line, 1024;
        if ($bytesread) {
          $hold_output[$fno][2] .= $line;
          $hold_output[$fno][1] += $line =~ y/\n/\n/;
          if ($hold_output[$fno][1] >= $g_maxlines) {
            $select->remove($handle);
          }
        } elsif ($!) {
          die "$!";
        } else {
          $hold_output[$fno][0] = 1; #EOF
          $select->remove($handle);
        }
      }
    }
    alarm(0);
  };
  if ($@) {
    # print STDERR "\$\@ = $@\n";
    die unless $@ eq "alarm\n";
  }

  # Note: lines may exceed $g_maxlines because of buffering
  # of output in the child process
  @$outref = split /\n/, $hold_output[fileno($childout)][2];
  @$errref = split /\n/, $hold_output[fileno($childerr)][2];
}

[0xbeef]

Okay, thanks for that! I would like however like to be enlightened more: ;)

1. I'd rather not use a solution using external modules, since the collector will have to run on MANY hosts as painlessly and non-intrusively as possible. Is there a high performance approach without external modules?
2. I am unsure as to how much data should be read from the pipe at a time in order to optimise throughput as much as possible...


-0xbeef

[graff]

1. I'd rather not use a solution using external modules, since the collector will have to run on MANY hosts as painlessly and non-intrusively as possible. Is there a high performance approach without external modules?

The modules cited in the code that Thelonius posted (IPC::Open3, IO::Select, Symbol) are not "external", in the sense that they are all included in the standard Perl "core" distribution. That is, wherever a reasonably current version of Perl is installed, these modules are also installed by default.

(If these many hosts you speak of have non-standard or hopelessly outdated perl installations, that's going to be a problem anyway.)

As for buffer size, the 1024 bytes suggested by Thelonius is fine for handling text that must ultimately be treated in a line-oriented fashion; if you're dealing with really high data rates, an 8k buffer should be about optimal.