Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.
 
PerlMonks  

Run N similar tasks in parallel

by bronto (Priest)
on Feb 20, 2003 at 14:14 UTC ( [id://237087]=CUFP: print w/replies, xml ) Need Help??

I needed a script that launched three instancies of the nmap network scanner to scan all our network in parallel. Each time a subnet is scanned and nmap finishes, another instance gets started to scan another subnet. This way I always have three parallel processes that do the job.

To learn how to do it I created this small script that could be a starting point for accomplishing similar tasks. You can substitute the sleep with an exec that runs an external program, for example, or fill that space with any perl you like!

Update: Thanks to other monks' suggestions I added "sanity checks": now the return value from fork and wait is checked. Thanks also to larsen, valdez, abell and alloalex's for their suggestions

#!/usr/bin/perl # forktest.pl # # I needed a script that launched three instancies of the nmap # network scanner to scan all our network in parallel. Each time # a subnet is scanned and nmap finishes, another instance gets # started to scan another subnet. This way I always have three # parallel processes that do the job. # # To learn how to do it I created this small script that could # be a starting point for accomplishing similar tasks. use strict ; use warnings ; my %childpid ; my $maxchild = 3 ; my $subnet = 1 ; do { while (keys(%childpid) < $maxchild and $subnet <= 254) { my $pid = fork ; # The following instruction comes from node 237098 die "Cannot fork: $!" unless defined $pid ; if ($pid) { # Parent process print "Child $pid started to scan subnet $subnet\n" ; $childpid{$pid} = $subnet++ ; } else { # Child process sleep rand 5 ; exit ; } } my $dead = wait ; # This die added after merlyn's suggestion # If you want to check a waitpid solution, see the code # posted by merlyn and zentara die "Something weird happened while wait!" if $dead == -1 ; print "Subnet $childpid{$dead} scan completed by child $dead\n" ; delete $childpid{$dead} ; } while (keys(%childpid) > 0) ;

Replies are listed 'Best First'.
•Re: Run N similar tasks in parallel
by merlyn (Sage) on Feb 20, 2003 at 14:24 UTC
    Here's the pattern I used for that on a recent project -- it's a bit more robust than yours.
    use POSIX ":sys_wait_h"; my @tasks = (1..398); my %kids; { while (@tasks and keys %kids < 5) { $kids{fork_a_task(shift @tasks)} = "active"; } { my $pid = waitpid(-1, 0); if ($pid == -1) { %kids = (); } else { delete $kids{$pid}; } } redo if @tasks or %kids; } sub fork_a_task { my $i = shift; my $pid = fork; return $pid if $pid; unless (defined $pid) { warn "cannot fork: $!"; return 0; } ## do stuff for task $i goes here... exit 0; }

    -- Randal L. Schwartz, Perl hacker
    Be sure to read my standard disclaimer if this is a reply.

      ++merlyn for checking fork's return value, that I forgot.

      Please, can you explain why your approach is more robust? I am really interested in it!

      Ciao and thanks!
      --bronto


      The very nature of Perl to be like natural language--inconsistant and full of dwim and special cases--makes it impossible to know it all without simply memorizing the documentation (which is not complete or totally correct anyway).
      --John M. Dlugosz
        By checking the possible failure of fork, I never "think" I'm working on five kids when only two have really started.

        By checking the value of waitpid for a possible -1 return, I never "think" I have kids left when I really don't. The -1 force-resets the hash.

        -- Randal L. Schwartz, Perl hacker
        Be sure to read my standard disclaimer if this is a reply.

Re: Run N similar tasks in parallel
by zentara (Archbishop) on Feb 21, 2003 at 12:50 UTC
    This work by abigail is one of my favorite pieces of code. I couldn't find the original node, so I include it here.
    #!/usr/bin/perl #by abigail #Some times you have a need to fork of several children, but you want +to #limit the maximum number of children that are alive at one time. Here + #are two little subroutines that might help you, mfork and afork. They + are very similar. #They take three arguments, #and differ in the first argument. For mfork, the first #argument is a number, indicating how many children should be forked. +For #afork, the first argument is an array - a child will be #forked for each array element. The second argument indicates the maxi +mum #number of children that may be alive at one time. The third argument +is a #code reference; this is the code that will be executed by the child. +One #argument will be given to this code fragment; for mfork it will be an + increasing number, #starting at one. Each next child gets the next number. For afork, the + array element is #passed. Note that this code will assume no other children will be spa +wned, #and that $SIG {CHLD} hasn't been set to IGNORE. mfork (10,10,\&hello); sub hello{print "hello world\n";} print "all done now\n"; ################################################### sub mfork ($$&) { my ($count, $max, $code) = @_; foreach my $c (1 .. $count) { wait unless $c <= $max; die "Fork failed: $!\n" unless defined (my $pid = fork); exit $code -> ($c) unless $pid; } 1 until -1 == wait; } ################################################## sub afork (\@$&) { my ($data, $max, $code) = @_; my $c = 0; foreach my $data (@$data) { wait unless ++ $c <= $max; die "Fork failed: $!\n" unless defined (my $pid = fork); exit $code -> ($data) unless $pid; } 1 until -1 == wait; } #####################################################
Re: Run N similar tasks in parallel
by tphyahoo (Vicar) on Oct 25, 2006 at 16:03 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: CUFP [id://237087]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others meditating upon the Monastery: (6)
As of 2024-03-19 02:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found