Re: Using forks and threads

[BrowserUk]

There is a lot of misinformation around.

perlthrtut was originally written about/relating to perl5005 threads, and has barely been revised since their demise in favour of iThreads.

Further, a lot of the casual advise available in reference to concurrency in Perl is dispensed by people who have made little or no attempt to utilise iThreads in any meaningful way, and is often coloured by experiences born of early 5005threading over early versions of *nix pthreads libraries, before they had figured out how to properly resolve threading and forking.

Using forking and threading in the same program does complicate matters. For example, what should happen if you call fork within a process that has 3 running threads?

• Should the new process be a complete clone of the entire process, including all 3 threads of the original?
• Or should the new process only be a clone of the thread that called fork?.

Under win32, calling system within a thread process is a perfectly safe thing to do as it doesn't use fork, or attempt to perform any cloning of the original process.

On other systems you will need to understand the function and limitations of the underlying OS threading libraries, and their interaction with the iThread implementation on your systems. There seems to be little or no information available regarding this in conjunction with *nix systems.

Basically, Perl threading has been so deamonised, that there are many areas of it's operation that have never been explored. If you are writing code for immediate production use, then you should probably avoid using a threads + fork combination as you would be trailblazing and may well encounter latent bugs and need to wait for several builds before fixes would become available.

If your program is less urgent and you have the will to explore the possibilities, especially if you would then be able to pass on your experiences, it's possible that the combination could be used to good effect and could benefit the wider community.

In either case, if you posted a description of the program you are going to write, and where you intend to use threads and fork, then you may well get some good advise on one or more ways of tackling the problem both with and without using threads.

[ColinHorne]

Many thanks for your reply, BrowserUk.

My project is neither urgent (I've been procrastinating for months now), nor mission critical, so I'm free to experiment and fiddle.

The project (named 'pancakes' due to the number of pancakes I consumed when I first had the idea) is a playlist framework 'thingy'. I'm still not sure exactly how pancakes will work, since I'm only in the early design phase, but I'm assuming that there will be forking/threading going on (typical setup of 1 thread for user interface, and 1 thread for doing the work).

I'm probably going to interact with mplayer (or perhaps XMMS2, if (stable) perl bindings ever get released for it). Not too keen on the alternatives (mpg123, etc), since they seem to only play mp3s/oggs (my playlist has all sorts of file formats in it). Interaction with mplayer will probably be done via system() or similar.

That's all I can really think of pancakes-wise (if anyone is interested and/or wants to help, feel very free to reply :-) ). It'll all be released under GPL, so whatever solution I find will be free for all. I'll probably also blog/wiki it (and if I make a good article, will post at the monastry).

From what has been said so far, it sounds like forking in a thread is unsafe, unportable, and unwise. I think the best thing in the given situation would be:

• fork the process
• child waits for instructions from parent
• parent does stuff (starts threads, etc)
• when the worker thread needs to call system(), it contacts child from step 1, and tells it to fork/exec, and send output back via pipe/socket
• etc

If I come up with any better ideas, I'll post another message :-)

Thanks to all who replied

[BrowserUk]

You really should read up on the pthreads libraries for your system, or perform a few experiments, to determine what happens when you fork a multi-threaded process. Personally I favour the latter approach as it generally yields quicker results.

A quick scan around suggests that POSIX/Solaris pthreads libraries map fork(2) to forkl(2) which only clones the calling thread, in which case it *ought* to be safe to use fork and threads in the same process which would greatly simplify what you describe. However, sucking and seeing is the only way to be sure.

If Perl's fork isn't compatible with threads on your system and you need to resort to forking early and running your system calls in a separate process, then I don't see much value in using threads. Using one process to run the user interface, and another to run the background processing would seem to be sufficient. Adding a second thread in the first process would simply complicate things as far as I can see.

[ColinHorne]

I'd just finished writing threads::system when I read your reply.

It's good news that fork seems to only clone the current thread, but, like you say, it sounds very system-dependant, and inconsistent.

May I ask for criticism on the following module? If enough people think it's worthwhile, I'll tweak it a bit, and then upload to CPAN (although, I may change the name to something less dominant first):

#!/usr/bin/perl

package threads::system;

require Exporter;
@ISA = qw{Exporter};
@EXPORT_OK = qw{system};

use threads::shared;
use IO::Handle;
use strict; use warnings;

pipe my ($crh, $cwh) or die "Could not pipe: $!";  # Command string re
+ad/write handles
pipe my ($erh, $ewh) or die "Could not pipe: $!";  # Exit status read/
+write handles
$cwh->autoflush(1); $ewh->autoflush(1);

my $sync : shared;

my $child = fork;
die "Could not fork: $!" if $child < 0;
unless ($child) {  # We are the child (daemon) - we handle system() ca
+lls for our parent
    close $erh; close $cwh;
    while (<$crh>) {  # Read commands to exec
        my $child = fork;
        die "Could not fork: $!" if $child < 0;
        unless ($child) {  # We are the child - we exec() for our pare
+nt
            exec $_; exit $!;
        } else {  # Parent - communicate our child's failure with our 
+own parent
            while (my $pid = waitpid $child, 0) {  # Is all this reall
+y needed?
                if ($pid < 0) { die "Error: waitpid: $!" }
                elsif ($pid == 0) { next }
                else { print $ewh $?, "\n"; last }  # Tell parent!
            }
        }
    }
    # Decide: is the following a feature or a bug?
    exit;  # Should never happen, unless client does system(undef), or
+ something
} else {  # Parent
    close $ewh; close $crh;
}

sub system {
    lock($sync);
    my $string = shift;  # TODO: Accept a full list
    print $cwh $string, "\n" or die "Could not write: $!";
    return <$erh>;
}

1

Example of usage:

#!/usr/bin/perl

use threads::system qw{system};

use strict; use warnings;

my $code = system("ls -al /etc");
print "Exited with code: $code\n";

Output is as expected. Thanks