Handling program output in real time

[dvergin]

Given some program that produces output over time, let's call it "blackbox.pl":

#!/usr/bin/perl
use warnings;
use strict;

for my $step (1..4) {
   sleep 3;
   print "Step $step\n";
}

I had expected (and wanted) the following code to capture and print out that output as it occurred:

#!/usr/bin/perl
use warnings;
use strict;

$|++;
open PIPE1, "./blackbox.pl |";
while ( <PIPE1> ) {
   # ...Filter and munge $_
   print $_;
}
close PIPE1;

But what it does (with or without the $|++) is wait until blackbox.pl finishes and then it prints out all the output at once.

What am I doing wrong? How can I capture and print the output of blackbox.pl as it occurs? (Needless to say, the real program will be doing some filtering and munging along the way.)

TIA, David

[pg]

Here is the fact behind what you observed: there are actually two types of buffering mechanism: line buffering and block buffering.

When you print to STDOUT:

• The output is line buffered, if the target is a terminal; (This why you see output every 3 seconds on the screen, if you run your blackbox.pl on its own.)
• otherwise (including pipe) the output is block buffered. (So in the pipe case, the problem is not your monitor program. But your blackbox.pl no longer outputs at the same pace, because of block buffering.)

By setting $| as Anonymous Monk indicated, you force the output to be flushed after each write.

[Anonymous Monk]

You need to set $| = 1; in "blackbox.pl".

[dvergin]

Yes, that *does* produce the desired behavior. (Interesting that the output from blackbox.pl spits out over time at the command line but not when it is piped. Hmmm...)

Unfortunately the actual program I am trying to track is pilot-xfer which reports its progress incrementally at the command line. And I have no control over its output buffering or whatever. It is what it is. (Thus the name of the stand-in: "blackbox".)

So where does that leave me in trying to solve my problem in the monitor program (the second one given)?

[Anonymous Monk]

IO::Pty may help you (system dependent), if your monitor program can emulate a terminal (tty) the xfer program will give it line buffered output (which is what you want).

[hmerrill]

Here is *maybe* another way to get what you want - not really sure about this, but here it is. How about having blackbox.pl write to a fifo (named pipe), and then having the script above read from the fifo? All you need to do is:

  1. create the fifo using 'mkfifo', and make sure blackbox.pl
     has write permissions to it.
  2. change blackbox.pl to write to the fifo file, instead
     of writing to STDOUT.
  3. change your script above to read from the fifo file
     instead of from PIPE1, like this

        my $fifo_file = "/path/to/my_fifo";
        open FIFO, "<$fifo_file" or die "Can't read $fifo_file: $!";
        while (<FIFO>) {
             # ...Filter and munge $_
             print $_;
        }

You'll have to play with this a little to get it to work the way you want, but basically written this way, this script(reading from FIFO) will block on the while (<FIFO>) until something is written to the fifo file.

Just for some more reference, the Perl Cookbook pages 576-580 has a recipe for working with a fifo. Also, Programming Perl 3rd Edition p.433 describes Named Pipes.

HTH.

[etcshadow]

This doesn't really address his issue, though, as writing to a FIFO is going to have the exact same buffering behavior (line based or block based) as writing to a pipe would.

Basically, as other users mentioned, perl internally does something along the lines of (psuedo-code):

if (-t STDOUT) {
   line_buffer STDOUT;
} else {
   block_buffer STDOUT;
}

Not that FIFOs aren't useful, they just don't have anything to do with this particular problem. If, for example, he had no control over the output of this "blackbox" and it wanted to write to a named file... well then a FIFO would be really useful, because he could replace the named file with a named pipe (FIFO), and all would be cool. Here, though, the guy's already got an anonymous pipe... so swapping it with a named-pipe isn't gonna really change anything.