real time output

[echoangel911]

Hi Monk, I have been using system($cmd) and `$cmd`; I have been trying to get the output of the command in real time. However, I been noticing that print `$cmd` will only print to screen when it's finished. For me, that takes 10 - 30 minutes. How can I print it in real time?

[liverpole]

Hi echoangel911,

I have to slightly disagree with both of the answers above.

If the process you're trying to read output from is under your control (eg. a Perl script), then Grandfather's advice will work, for the program doing the output.

However, if you are getting the output from some other program which you don't have the source code to, you may be out of luck.  This is because the system is probably buffering the output; saving up the output until a certain fixed amount is reached, after which it will send the entire buffer.  The only way the process will send the a small amount of data is when the process finishes, and it closes its output file, which will cause the buffer to flush.

Unless you have control over the flushing mechanism of the program producing the output, you'll just have to be patient until the process finishes.

Here's an example.  Let's say that you *do* have the source to the first program (the one producing the output), and it's call proc1.pl:

#!/usr/bin/perl -w

use strict;
use warnings;

for (my $i = 1; $i <= 60; $i++) {
    sleep 1;
    print "$i\n";
}

Now, here's hgolden's program (modified slightly to avoid global variables, and to die if the pipe open failed).  The second program, proc2.pl looks like this:

#!/usr/bin/perl -w

use strict;
use warnings;

my $cmd = "proc1.pl";
open(my $pipefh, "$cmd |") or die "Unable to pipe to command '$cmd' ($
+!)\n";
my $out;
while ($out = <$pipefh>) {
    print $out;
}

You will likely see, when you run proc2.pl, that no output is produced until proc1.pl finishes, which will take 60 seconds.  That's because the output is buffered until a certain amount is written.  The way to fix it is by modifying proc1.pl to set autoflush true as Grandfather described:

#!/usr/bin/perl -w

use strict;
use warnings;

$| = 1;

for (my $i = 1; $i <= 60; $i++) {
    sleep 1;
    print "$i\n";
}

Now, you should see output from proc1.pl every second.

[sgifford]

That's a good point liverpole. It's useful to know that the default buffering policy is often controlled by whether the process is writing to a tty or not. You can persuade many programs to line-buffer their output by making them think they're writing to a TTY, using a module like Expect or IO::Pty.

[echoangel911]

thanks for your help! it was very insightfull

[GrandFather]

Have you tried setting $| (auto flush) true? Perl flushes output after every print then rather than buffering until it has a nice pile of stuff to output.

[hgolden]

Hey

I may be wrong, but I suspect the issue isn't the buffering (though you should try that too), but is instead that you aren't reading the output before the command finishes. Depending on what you're doing, you may be able to do a piped open on the command, and capture the output that way. That would go something like this:

open(COMMAND, "$cmd |");
while($out=<COMMAND>) {
print "$out";
}

Check out more about Open at http://perldoc.perl.org/functions/open.html and let me know if you're having trouble.

[zentara]

Here is a simple example which you can play with....IPC3 buffer limit problem

The program 'bc' will act like this, and will seem to chop off the last buffer of output. There is a way to detect what is in the buffer, and suck it out. It does have a flaw, as noted by merlyn. Anyways, this is documented in "perldoc -q waiting"