Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Handling program output in real time

by dvergin (Monsignor)
on Nov 10, 2003 at 06:26 UTC ( [id://305801]=perlquestion: print w/replies, xml ) Need Help??

dvergin has asked for the wisdom of the Perl Monks concerning the following question:

Given some program that produces output over time, let's call it "":
#!/usr/bin/perl use warnings; use strict; for my $step (1..4) { sleep 3; print "Step $step\n"; }
I had expected (and wanted) the following code to capture and print out that output as it occurred:
#!/usr/bin/perl use warnings; use strict; $|++; open PIPE1, "./ |"; while ( <PIPE1> ) { # ...Filter and munge $_ print $_; } close PIPE1;
But what it does (with or without the $|++) is wait until finishes and then it prints out all the output at once.

What am I doing wrong? How can I capture and print the output of as it occurs? (Needless to say, the real program will be doing some filtering and munging along the way.)

TIA, David

"Perl is a mess and that's good because the
problem space is also a mess.
" - Larry Wall

Replies are listed 'Best First'.
Re: Handling program output in real time
by pg (Canon) on Nov 10, 2003 at 07:55 UTC

    Here is the fact behind what you observed: there are actually two types of buffering mechanism: line buffering and block buffering.

    When you print to STDOUT:

    • The output is line buffered, if the target is a terminal; (This why you see output every 3 seconds on the screen, if you run your on its own.)
    • otherwise (including pipe) the output is block buffered. (So in the pipe case, the problem is not your monitor program. But your no longer outputs at the same pace, because of block buffering.)

    By setting $| as Anonymous Monk indicated, you force the output to be flushed after each write.

Re: Handling program output in real time
by Anonymous Monk on Nov 10, 2003 at 06:31 UTC
    You need to set $| = 1; in "".
      Yes, that *does* produce the desired behavior. (Interesting that the output from spits out over time at the command line but not when it is piped. Hmmm...)

      Unfortunately the actual program I am trying to track is pilot-xfer which reports its progress incrementally at the command line. And I have no control over its output buffering or whatever. It is what it is. (Thus the name of the stand-in: "blackbox".)

      So where does that leave me in trying to solve my problem in the monitor program (the second one given)?

        IO::Pty may help you (system dependent), if your monitor program can emulate a terminal (tty) the xfer program will give it line buffered output (which is what you want).
Re: Handling program output in real time
by hmerrill (Friar) on Nov 10, 2003 at 14:29 UTC
    Here is *maybe* another way to get what you want - not really sure about this, but here it is. How about having write to a fifo (named pipe), and then having the script above read from the fifo? All you need to do is:
    1. create the fifo using 'mkfifo', and make sure has write permissions to it. 2. change to write to the fifo file, instead of writing to STDOUT. 3. change your script above to read from the fifo file instead of from PIPE1, like this my $fifo_file = "/path/to/my_fifo"; open FIFO, "<$fifo_file" or die "Can't read $fifo_file: $!"; while (<FIFO>) { # ...Filter and munge $_ print $_; }

    You'll have to play with this a little to get it to work the way you want, but basically written this way, this script(reading from FIFO) will block on the while (<FIFO>) until something is written to the fifo file.

    Just for some more reference, the Perl Cookbook pages 576-580 has a recipe for working with a fifo. Also, Programming Perl 3rd Edition p.433 describes Named Pipes.

      This doesn't really address his issue, though, as writing to a FIFO is going to have the exact same buffering behavior (line based or block based) as writing to a pipe would.

      Basically, as other users mentioned, perl internally does something along the lines of (psuedo-code):

      if (-t STDOUT) { line_buffer STDOUT; } else { block_buffer STDOUT; }
      Not that FIFOs aren't useful, they just don't have anything to do with this particular problem. If, for example, he had no control over the output of this "blackbox" and it wanted to write to a named file... well then a FIFO would be really useful, because he could replace the named file with a named pipe (FIFO), and all would be cool. Here, though, the guy's already got an anonymous pipe... so swapping it with a named-pipe isn't gonna really change anything.

      Not an editor command: Wq

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://305801]
Approved by Roger
Front-paged by bart
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others wandering the Monastery: (6)
As of 2024-04-13 19:44 GMT
Find Nodes?
    Voting Booth?

    No recent polls found