Interacting with the shell ( backticks, system(), open -| )

[jktstance]

Hi again, I have this code from a script I want to modify: open FP, "find . -ls |" From what I understand, this will pipe the output of find() to the FP filehandle. Then you can loop through it like a standard text file:

while (my $file = <FP>) {
    chomp $file;
    ...
}

What is the advantage of using open FP, "find . -ls |" and then looping through the handle, as opposed to using backticks?

my @files = `find . -ls`;
foreach my $line (@lines) {
   chomp $file;
   ...
}

Also, regarding system(), it returns the exact same thing as using backticks, plus the final element of @files is the exit status (0 in this case for success). I thought system() was only supposed to return an exit status, not full output TL;DR -- why did the coder use a pipe in open() instead of using backticks or system()? FYI, this script basically reads the find . -ls output and writes a file listing files and their sizes.

[BrowserUk]

What is the advantage of using open FP, "find . -ls |" and then looping through the handle, as opposed to using backticks?

1. If the output is extensive, reading and processing it a line at a time rather than slurping it all into memory then processing it, may avoid using a large amount of memory.
2. If the output takes a while to produce, your program can be processing the first lines whilst the other process is still producing the rest.
3. Maybe you don't need all the output.

   Once you found what you are looking for, you can discard the rest.

   If you capture the pid returned by the piped open, you can even kill the external process before it completes.

regarding system(), it returns the exact same thing as using backticks

What makes you think this? system does not capture or return any output from the program.

why did the coder use a pipe in open() instead of using backticks ...?

Perhaps he was dealing with very large directories.

[jktstance]

Thank you. So piping the command to a filehandle should keep the find . -ls process open until <FP> is closed? Anyway, it makes more sense to pipe if it doesn't slurp everything up at once like assigned the command to an array.

I also responded to davido's reply above as to why system() is returning the output of the command, in addition to the exit status.

[BrowserUk]

So piping the command to a filehandle should keep the find . -ls process open until <FP> is closed?

The external process is connected to your end of the pipe via one or two buffers, usually 4k or 8k each.

When (if) those buffers fill up, the external process will be blocked when trying to add more output to the pipe and thus will have to wait until your program reads some of the data from your end of the pipe before continuing.

If the external program produces less output than the combined size of the buffers, it will run to completion immediately and the output will wait in the pipe until your program reads it.

[davido]

I don't see your code where a call to system produces the same output as backticks. Can you provide an example that we can execute ourselves to see what you mean?

The equivalence of `find . -ls`; and open FP, 'find . -ls |' is kind of where the functional overlap ends. Backticks are more confining than using open, and while you can emulate what backticks do using open, you cannot emulate everything that open can do using backticks. And with backticks you get everything back at the same time. In the simple example you presented, one of the best reasons to go with the open version is to allow you to work on the output one line at a time, without storing it all in an array. And the advantage of the backticks in this same example is just the simplicity and brevity.

[jktstance]

Dave, Here is the script I use when using system():

#! /usr/bin/perl
use strict;
use warnings;

my @files = system "find . -ls";

foreach my $file (@files) {
    chomp $file;
        print "$file\n";
}

And it outputs the following (I deleted the userid column):

1105134059    2 drwxr-xr-x   3 .... users        2048 Feb  5 09:40 .
1105135378    1 -rw-r--r--   1 .... users         506 Feb  3 10:54 ./t
+est.pl
1105135090    1 -rw-r--r--   1 .... users          93 Jan 24 15:41 ./c
+alendar.pl
0

Notice the trailing 0? If I replace my @files = system "find . -ls"; with my @files = `find . -ls`;, I get the same output, except the trailing 0 is missing.

I understand why system() prints out the trailing 0 (it's the exit status of find()), but why is it also outputting everything find() outputs? I thought it should only return the exit status.

[choroba]

find itself outputs the lines. System does not return them, it only returns the exit status. You can verify none of the lines is printed via print by wrapping the values in parentheses:

    print "($file)\n";

The zero will be the only parenthesized string.

[Corion]

Have you compared system against qx? They do different things. Most importantly, system does not return the output of the child.

[Discipulus]

Only one word to add to previous, wise, answers: iterator. The diamond operator around your filehandle is an iterator.
High Order Perl book has a wonderful chapter about this and an implementation can be also found on Cpan.
When you search a number in your address book you prefere to learn all entries by memory and only after the last number dial the number you searched for (backticks), or descend the list until you find the number (iterator)?
hth
L*
UPDATE: link to the book as suggested by AnomalousMonk below.

[AnomalousMonk]

Free download for HOP here.

[afoken]

why did the coder use a pipe in open() instead of using backticks or system()?

An even better question: Why didn't (s)he use File::Find? There is no need to spawn a child process just to find files in a directory tree.

Consider rewriting that code to use File::Find.

Alexander

[jktstance]

How could I use File::Find and still keep the feature of the pipe into a filehandle, namely the while loop only reading in one line at a time? As with using backticks, the only way I can see is to read the entire output of File::Find into an array, which could be very large.

[afoken]

How could I use File::Find and still keep the feature of the pipe into a filehandle, namely the while loop only reading in one line at a time? As with using backticks, the only way I can see is to read the entire output of File::Find into an array, which could be very large.

Do you know File::Find? If not, did you read the documentation?

My guess is that your answers are NO and NO, and the second one makes me sad.

File::Find has no output. File::Find uses a callback function (commonly called wanted) that is invoked exactly once for every file found. The loop that traverses the directory tree, the equivalent of the pipe open and the while / readline loop, is inside the File::Find::find funcion, so you don't need the while loop any more. The equivalent of the loop body is the callback function, this is where you process any file found by File::Find. You do not need an array.

Consult the really good documentation of File::Find for details.

Alexander