Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Handling program output in real time

by dvergin (Monsignor)
on Nov 10, 2003 at 06:26 UTC ( #305801=perlquestion: print w/ replies, xml ) Need Help??
dvergin has asked for the wisdom of the Perl Monks concerning the following question:

Given some program that produces output over time, let's call it "blackbox.pl":
#!/usr/bin/perl use warnings; use strict; for my $step (1..4) { sleep 3; print "Step $step\n"; }
I had expected (and wanted) the following code to capture and print out that output as it occurred:
#!/usr/bin/perl use warnings; use strict; $|++; open PIPE1, "./blackbox.pl |"; while ( <PIPE1> ) { # ...Filter and munge $_ print $_; } close PIPE1;
But what it does (with or without the $|++) is wait until blackbox.pl finishes and then it prints out all the output at once.

What am I doing wrong? How can I capture and print the output of blackbox.pl as it occurs? (Needless to say, the real program will be doing some filtering and munging along the way.)

TIA, David

------------------------------------------------------------
"Perl is a mess and that's good because the
problem space is also a mess.
" - Larry Wall

Comment on Handling program output in real time
Select or Download Code
Re: Handling program output in real time
by Anonymous Monk on Nov 10, 2003 at 06:31 UTC
    You need to set $| = 1; in "blackbox.pl".
      Yes, that *does* produce the desired behavior. (Interesting that the output from blackbox.pl spits out over time at the command line but not when it is piped. Hmmm...)

      Unfortunately the actual program I am trying to track is pilot-xfer which reports its progress incrementally at the command line. And I have no control over its output buffering or whatever. It is what it is. (Thus the name of the stand-in: "blackbox".)

      So where does that leave me in trying to solve my problem in the monitor program (the second one given)?

        IO::Pty may help you (system dependent), if your monitor program can emulate a terminal (tty) the xfer program will give it line buffered output (which is what you want).
Re: Handling program output in real time
by pg (Canon) on Nov 10, 2003 at 07:55 UTC

    Here is the fact behind what you observed: there are actually two types of buffering mechanism: line buffering and block buffering.

    When you print to STDOUT:

    • The output is line buffered, if the target is a terminal; (This why you see output every 3 seconds on the screen, if you run your blackbox.pl on its own.)
    • otherwise (including pipe) the output is block buffered. (So in the pipe case, the problem is not your monitor program. But your blackbox.pl no longer outputs at the same pace, because of block buffering.)

    By setting $| as Anonymous Monk indicated, you force the output to be flushed after each write.

Re: Handling program output in real time
by hmerrill (Friar) on Nov 10, 2003 at 14:29 UTC
    Here is *maybe* another way to get what you want - not really sure about this, but here it is. How about having blackbox.pl write to a fifo (named pipe), and then having the script above read from the fifo? All you need to do is:
    1. create the fifo using 'mkfifo', and make sure blackbox.pl has write permissions to it. 2. change blackbox.pl to write to the fifo file, instead of writing to STDOUT. 3. change your script above to read from the fifo file instead of from PIPE1, like this my $fifo_file = "/path/to/my_fifo"; open FIFO, "<$fifo_file" or die "Can't read $fifo_file: $!"; while (<FIFO>) { # ...Filter and munge $_ print $_; }

    You'll have to play with this a little to get it to work the way you want, but basically written this way, this script(reading from FIFO) will block on the while (<FIFO>) until something is written to the fifo file.

    Just for some more reference, the Perl Cookbook pages 576-580 has a recipe for working with a fifo. Also, Programming Perl 3rd Edition p.433 describes Named Pipes.

    HTH.
      This doesn't really address his issue, though, as writing to a FIFO is going to have the exact same buffering behavior (line based or block based) as writing to a pipe would.

      Basically, as other users mentioned, perl internally does something along the lines of (psuedo-code):

      if (-t STDOUT) { line_buffer STDOUT; } else { block_buffer STDOUT; }
      Not that FIFOs aren't useful, they just don't have anything to do with this particular problem. If, for example, he had no control over the output of this "blackbox" and it wanted to write to a named file... well then a FIFO would be really useful, because he could replace the named file with a named pipe (FIFO), and all would be cool. Here, though, the guy's already got an anonymous pipe... so swapping it with a named-pipe isn't gonna really change anything.

      ------------
      :Wq
      Not an editor command: Wq

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://305801]
Approved by Roger
Front-paged by bart
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (14)
As of 2014-08-01 17:04 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Who would be the most fun to work for?















    Results (33 votes), past polls