What is a pipe and why would I want to use one?

[whiteperl]

I guess that you could say that i'm fairly new to programing so i'd like to findout what exactly are pipes and what do you do with them? whiteperl

[fokat]

A pipe is a form of Inter-Process Communication or IPC. Think of it as an actual pipe that takes the output of a process and "pipes it" to the input of another process.

In most operating systems that support this mechanism, the pipes are implemented as file descriptors that one of the processes writes to and the other, reads from.

Pipes are very useful in producer - consumer problems. In Unix you'll see a lot of that, when you need to combine two commands to perform a particular action. For instance, in order to get the list of files in your current directory, sorted by lexicographical order, you would issue the following shell command:

ls | sort

When handling this command, the shell will arrange for the process "ls" to run, sending its output to a pipe and the process "sort" to take its input from the same pipe. Therefore, every byte produced by the "ls" command will be eventually read and processed by "sort".

Perl has the capability to use pipes. It can automatically send your program's output to a pipeline by saying something like

open(FHANDLE, "| sort")

Or get input from a pipeline, such as in

open(FHANDLE, "ls |")

You also have "low level" control of the pipes. You can create a pipe, fork() and use it to keep father and child (or siblings or whatever) happily talking to each other. The call to do this is (surprisingly) called pipe. See perldoc for more details and perlipc for an excellent tutorial on the different forms of IPC available on Perl.

Hope this helps.

[rjray]

Excellent explanation of the concepts, well worth a ++.

You also have "low level" control of the pipes. You can create a pipe, fork() and use it to keep father and child (or siblings or whatever) happily talking to each other. The call to do this is (surprisingly) called pipe.

I would also recommend having people look at the IO::Pipe class that is a part of the core. It's very easy to use, and integrates nicely with IO::File, IO::Socket, et al.

--rjray

[Zaxo]

A pipe is a method of Interprocess Communication (IPC). It supports moving data from one process to another by creating a pair of filehandles, one write-only, one read-only. The pipe must be created before fork so that the filehandles are known to both processes. The writer process closes the read-only handle, the reader closes the write handle. At that point, the writer prints whatever it has to communicate to the write handle, and the reader will find the data on the read handle.

I've seen plenty of good examples here, but here is one that does so little that the main features are easy to see:

#!/usr/bin/perl -w
use strict;

my ($in,$out,$pid);
pipe $in, $out or die $!;

defined( $pid = fork() ) or die "No fork: ", $!;
if ($pid == 0) { # in child
    close $out;
    print "From Child: ", $_ while <$in>;
    exit 0;
}
# in parent
close $in;
print $out "From Parent: ", $_ while <>;
close $out;
wait;

We close $out in the parent so that the child's print loop ends, and it exits. The parent waits for the child's exit so as to clean up afterwords.

There are lots of other ways of doing multiprocess programming and IPC. There is a form of the open function which takes care of the details for you.

I'd recommend Advanced Programming in the UNIX Environment for a good explanation of processes and ipc generally. perlipc is a concentrated course in using perl's process and ipc functions together.

After Compline,
Zaxo

[BrowserUk]

Pipes are a way of connecting to programs together as they run so that the output from the first is fed into the input from the second. If you have ever typed type readme.txt | more(+) (or perhaps cat readme | more(+) if your a linux user), then that vertical bar (|) is an instruction to the command line processor (CLP) to use an (anonynous*) pipe btween the program type (**) or cat and the program more. This way, instead of the output from type (cat) coming straight onto the screen (via STDOUT) it is fed into the more program (via STDIN). It's this connection that is known as a pipe.

On some older, single tasking operating systems (OS's), meaning that the could only run one program at a time. ie. DOS. The way pipes where implemented was that the DOS CLP (command.com) would open a temporary output file on disk and pass that to the type command for it to place it's output in. When the type program finished, the CLP would close and reopen the temporary file for input and pass this to the more program as STDIN. The more program would then read the temporary file and display it 24 or 34 lines at a time depending on how big your screen was. Luckily, this ludicrous situation is no longer true in DOSish (Windows) OS's and they do the sensible thing, as all unix OS's have always done (and even their forerunners, DEC RSTS/E and RSX did way back) and simply share a memory buffer between the two programs that they can write to and read from as if the pipe was a file.

This very simple concept is the very essense of the "unix way". It is the glue which allows lots of small programs that do one (usually simple) task very well to be combined together to perform lots of different functions.

I am sure that you will get lots of good clear examples of how the concept of pipes can be useful with Perl programs from the other guys that have much greater knowledge on that than I do.

Have fun with pipes. They are very useful.

+. Of course, this an inefficient way of pausing a file, more < readme.txt or even  more readme is much better, but it's only here by way of illustration.

*. On windows system's there are also things called named pipes. These can be used to connect between programs running on different machines. The way these work is somewhat more complicated, but the principles are much the same. To the programs running they simple look like files, and are read and written using the same calls as files.

**. Actually, on modern windows OS's, type isn't a seperate program stored in a .com or .exe file, its a built-in command within the CMD.exe CLP, but the effect amounts to the same thing.

[whiteperl]

Thanks for all the replys.
My only concern now is if I open a pipe to another program if the other program automatically opens.


Senario:


I open Program 1 - Daddy -> open(SEND,"|Kiddy.pl");

Program pipes its output to Program 2 - Kiddy

Does Kiddy automatically open?


Next where does Kiddy.pl get the data from Daddy.pl

Is it in STDIN as in -> read(STDIN, $from_Daddy, $ENV{CONTENT_LENGTH});

Is it in -> $from_Daddy = $ENV{QUERY_STRING}; #\ This seems unlikely but you never know /#

or do i get it like open(GET,"Daddy.pl|");

whiteperl

[rjray]

Using open to create the pipe for you in this fashion does in fact execute the child program, yes.

There are no environment variables involved, so none of your examples are exactly correct. When Kiddy.pl starts, it will have a STDIN that is not connected to the TTY, like Daddy.pl has. Instead, STDIN is like any other filehandle (which it is, anyway). You read from it with "$line = <STDIN>" just as you would with any other filehandle. Whatever the Daddy.pl writed to SEND, Kiddy.pl reads from STDIN.

Your third example is close, though. What it would do is run a separate execution of Daddy.pl when you manually ran Kiddy.pl. Then, whatever Daddy.pl wrote to STDOUT, Kiddy.pl would read from GET.

One last thing worth noting: if your program wants to know if a given filehandle is a pipe, you can test it with the -p file-test operator:

my $is_a_pipe = -p STDIN;

This is very useful when testing if the program is writing to a pipe-- if it is, you may choose to format data differently, or if you were pausing at page breaks you would choose not to, etc.

--rjray

[fsn]

This is a very very simple (and probably Linuxcentric) attempt to illustrate how you can send data from one program to another using a pipe:

./Daddy.pl:

#!/usr/bin/perl

$i=0;
open SESAME,"|./Kiddy.pl";      # Open up the pipe

while ( $i++<10) {
        print SESAME "Testing IPC with pipes. Line $i.\n";      # Prin
+t to pipe
}

close SESAME;    # This (or the termination of the program)
                 # closes the pipe. The other side recieves this
                 # as an EOF (End OF File).


./Kiddy.pl:

#!/usr/bin/perl

while ($in=<STDIN>) {                # Read from <STDIN> until <EOF>
        chomp $in;                   # Remove <NL>
        print "By Kiddy.pl: $in\n";   # Print out what we read
}


There are some extra fluff in Kiddy.pl, the use of chomp and $in instead of $_ f.ex., but I kept it there to make it more clear, at least to me.

If you run the above you would get this:

$ ./Daddy.pl
By Kiddy.pl: Testing IPC with pipes. Line 1.
By Kiddy.pl: Testing IPC with pipes. Line 2.
By Kiddy.pl: Testing IPC with pipes. Line 3.
By Kiddy.pl: Testing IPC with pipes. Line 4.
By Kiddy.pl: Testing IPC with pipes. Line 5.
By Kiddy.pl: Testing IPC with pipes. Line 6.
By Kiddy.pl: Testing IPC with pipes. Line 7.
By Kiddy.pl: Testing IPC with pipes. Line 8.
By Kiddy.pl: Testing IPC with pipes. Line 9.
By Kiddy.pl: Testing IPC with pipes. Line 10.
$


Note that Daddy.pl is not printing to the screen at all, Kiddy.pl is doing all that.

[rah]

Not a comment on the node, so much as a comment on the replies. Three cheers and ++ to all the monks who responded to whiteperl for their excellent answers. This inquiry could easily be dismissed as "not a perl question". Instead these monks stepped forward with thorough and thoughtful answers. I'm not new to pipes, though I have yet to use perl to wrestle with them, yet I still learned a few things. Great job monks!