capturing command output

[morgon]

Hi,

in some legacy code (running on Windows) we came across this to capture the output of an external command:

sub fetch_cmd {
    my ($cmd) = @_;
    my $ret;

    unless ( open FH, $cmd . ' 2>&1 |' ) {
        return;
    }
    while (<FH>) {
        $ret .= $_;
    }
    close FH;
    return $ret;
}

We are currently having some discussions on whether the above construct involves a shell (I think it does) and whether it is better then simply using backticks (I think it is not).

What is your opinion?

[BrowserUk]

It definitely does involve a shell; and I see no advantage, and several disadvantages over backticks.

[moritz]

I'm pretty sure it does involve a shell; what other part should take care of the redirection 2>&1?

And I don't think it's any better than backticks. It also lacks proper error handling.

[Eliya]

what other part should take care of the redirection 2>&1?

Perl itself?  Like it does on Unix when 2>&1 is the only shell meta characters in the command.^*

I don't think Perl is making this optimisation on Windows, though, but I do think it's a valid question — considering that in Perl things are not always quite as obvious as it might seem at first...

___

^* as you can easily verify:

$ strace -fqeexecve perl -e 'open FH, "echo foo bar 2>&1 |"; print <FH
+>'
execve("/usr/bin/perl", ["perl", "-e", "open FH, \"echo foo bar 2>&1 |
+\"; "...], [/* 77 vars */]) = 0
[pid  1819] execve("/bin/echo", ["echo", "foo", "bar"], [/* 77 vars */
+]) = 0
foo bar

No shell involved here.  But as soon as you add another shell metacharacter, e.g. ";", Perl will use the shell:

$ strace -fqeexecve perl -e 'open FH, "echo foo bar ; 2>&1 |"; print <
+FH>'
execve("/usr/bin/perl", ["perl", "-e", "open FH, \"echo foo bar ; 2>&1
+ |\""...], [/* 77 vars */]) = 0
[pid  1833] execve("/bin/sh", ["sh", "-c", "echo foo bar ; 2>&1"], [/*
+ 77 vars */]) = 0
foo bar

[chrestomanci]

Very interesting. ++

I had no idea that perl did that optimization, it is good to know, but it would be nice to have a way to control it.

[JavaFan]

Considering all you do is run a command, and collect its output and errors, and return a string with said output and errors, I see no advantage of not using backticks. The backticks will also use the shell, but that doesn't really matter. Your fetch_cmd basically is a reimplementation of backticks.

[chrestomanci]

The OP's script captures both STDOUT and STDERR, but plain backticks will only capture STDOUT.

As a demo, I used a script like this:

#!/usr/bin/perl
print STDOUT "hello\n";
print STDERR "world\n";

If you run it on a bash cmd line you get both lines of output as you would expect. If you call via perl backticks then only "hello" is captured, and "world" is still sent to stdout.

[JavaFan]

If you call via perl backticks then only "hello" is captured, and "world" is still sent to stdout.

I cannot reproduce that.

$ cat foo
#!/usr/bin/perl
print STDOUT "hello\n";
print STDERR "world\n";
$ cat bar
#!/usr/bin/perl
use 5.010;
my $output = `./foo 2>&1`;
say "[[$output]]";
$ chmod +x foo
$ chmod +x bar
$ ./bar
[[world
hello
]]
$

Or are you suggesting the OP would run a different command in backticks than in his open? Why would he do that?

[pklausner]

> Your fetch_cmd basically is a reimplementation of backticks.
Yes. But besides handling stderr it gives you another layer of indirection. And as the saying goes: there is not problem you cannot address with another layer of indirection ;-) Here it might actually be useful for logging the call (if it's generated) or storing the output for later inspection (it may not look as expected) or discard stderr or...

[JavaFan]

But besides handling stderr it gives you another layer of indirection.

So what? You have any idea how many layers of indirection there are on the open call? Or on the reading? Why is the number of layers of redirection an argument against backticks, but not about open, or the diamond operator?

Here it might actually be useful for logging the call (if it's generated) or storing the output for later inspection (it may not look as expected) or discard stderr or...

Sure, but he isn't doing any of that. One can critic any piece of code with "yeah, but if you want to do X in the future, it becomes harder". With arguments like that, you'd never write down a single line of code.

[Ratazong]

Two aspects you might want to consider

• writing to a temporary output file (as done in your example) might lead to troubles with write-access-rights
• writing to a temporary output file is helpful when debugging, as that file can be checked easily afterwards outside of perl (unless you delete it)
In my projects, I have no real preference. I typically decide it case-base-case based on the "feeled risk of needing debugging info"

HTH, Rata

Update: Oooops! I have overlooked the pipe-symbol. So please ignore my answer above! Thanks for pointing this out, BrowserUk!

[BrowserUk]

The code is not writing to a temporary file. The external process is writing directly to a pipe which is then read in the while loop.

[eyepopslikeamosquito]

We tend to avoid the Windows shell in Perl scripts that run external commands in customer environments over which we have no control. Some years ago we got burnt because some of our customers use a misfeature of Windows cmd.exe, namely that without the /d option, cmd.exe executes stuff from one of the following Registry values:

HKEY_LOCAL_MACHINE\Software\Microsoft\Command Processor\AutoRun
HKEY_CURRENT_USER\Software\Microsoft\Command Processor\AutoRun

before running your command. In one case, I remember there was a "cd" command in the customer autorun Registry which caused one of our Perl scripts to malfunction. We worked around it by replacing `some command 2>&1` with `cmd /d/c "some command" 2>&1`. An alternative workaround is to manipulate STDOUT and STDERR from perl, while taking care to call the Perl system function in such a way that the shell is not invoked.

To clarify, the two approaches are illustrated by the following two example programs:

use strict;
use warnings;
my $cmd       = 'perl -le "print qq{to stdout};print STDERR qq{to stde
+rr};exit 42"';
my $cmdhack   = qq{cmd /d/c $cmd 2>&1};    # works
# my $cmdhack = qq{cmd /d/c "$cmd" 2>&1};  # also works (despite bizar
+re quoting)
print "run:$cmdhack:\n";
my $out = `$cmdhack`;
my $rc = $? >> 8;
print "rc=$rc, stdout/stderr of command follows:\n";
print $out;

and:

use strict;
use warnings;
my $outfile = 'ff.tmp';
my $exe     = $^X;
my @args    = ( 'perl', '-le', 'print qq{to stdout};print STDERR qq{to
+ stderr};exit 42' );
print "run exe:$exe:args:@args:\n";
print STDERR "here we go to stderr\n";
open(SAVOUT, ">&STDOUT") or die "error: save original STDOUT: $!";
open(SAVERR, ">&STDERR") or die "error: save original STDERR: $!";
open(STDOUT, '>', $outfile) or die "error: create   '$outfile': $!";
open(STDERR, '>&STDOUT')    or die "error: redirect '$outfile': $!";
system { $exe } @args;
my $rc = $? >> 8;
open(STDOUT, ">&SAVOUT") or die "error: restore STDOUT: $!";
open(STDERR, ">&SAVERR") or die "error: restore STDERR: $!";
close(SAVERR) or die "error: close SAVERR: $!";
close(SAVOUT) or die "error: close SAVOUT: $!";
print "rc=$rc\n";
print STDERR "rc=$rc to STDERR\n";

Improvements welcome.

Update: tidied up and clarified the example code.