Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

•Re: Run N similar tasks in parallel

by merlyn (Sage)
on Feb 20, 2003 at 14:24 UTC ( #237089=note: print w/ replies, xml ) Need Help??


in reply to Run N similar tasks in parallel

Here's the pattern I used for that on a recent project -- it's a bit more robust than yours.

use POSIX ":sys_wait_h"; my @tasks = (1..398); my %kids; { while (@tasks and keys %kids < 5) { $kids{fork_a_task(shift @tasks)} = "active"; } { my $pid = waitpid(-1, 0); if ($pid == -1) { %kids = (); } else { delete $kids{$pid}; } } redo if @tasks or %kids; } sub fork_a_task { my $i = shift; my $pid = fork; return $pid if $pid; unless (defined $pid) { warn "cannot fork: $!"; return 0; } ## do stuff for task $i goes here... exit 0; }

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.


Comment on •Re: Run N similar tasks in parallel
Download Code
Re: &bull;Re: Run N similar tasks in parallel
by bronto (Priest) on Feb 20, 2003 at 14:47 UTC

    ++merlyn for checking fork's return value, that I forgot.

    Please, can you explain why your approach is more robust? I am really interested in it!

    Ciao and thanks!
    --bronto


    The very nature of Perl to be like natural language--inconsistant and full of dwim and special cases--makes it impossible to know it all without simply memorizing the documentation (which is not complete or totally correct anyway).
    --John M. Dlugosz
      By checking the possible failure of fork, I never "think" I'm working on five kids when only two have really started.

      By checking the value of waitpid for a possible -1 return, I never "think" I have kids left when I really don't. The -1 force-resets the hash.

      -- Randal L. Schwartz, Perl hacker
      Be sure to read my standard disclaimer if this is a reply.

        By checking the value of waitpid for a possible -1 return, I never "think" I have kids left when I really don't. The -1 force-resets the hash.

        Good, I should check for the return value of wait as well. Thanks.

        Sorry if I am bothering you about this, but I am missing an information: why do you prefer waitpid to wait? Flexybility? Possibility to make waitpid a non-blocking call? What?

        Thanks in advance.

        --bronto


        The very nature of Perl to be like natural language--inconsistant and full of dwim and special cases--makes it impossible to know it all without simply memorizing the documentation (which is not complete or totally correct anyway).
        --John M. Dlugosz

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://237089]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (4)
As of 2015-07-03 03:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (47 votes), past polls