Fork limited childs and passing parameters

[chanakya]

Dear Monks,

I'm trying to write a program which will calls a subroutine with a given sourceid.
There can be around 200 sourceids which are nothing but directory names.
The script should initially spawn 4 childs and call the subroutine with a
given sourceid (the sourceid cannot be repeated). Script should create new childs
as and when a child finishes the job and continue until all the 200 sourceids are processed.

Below is the code I've tried. My only question is
Is there anyway of doing the same thing in a simple manner and the most important thing
how do I pass the sourceids(unique) to the subroutine.

#!/usr/bin/perl -w
my $NumChilds = 4;

@sourceids=("aex", "athena", "athens","bidapa","bbic","bony","capctha"
+,"cooldo","italy","paris","xmltech");
my $sourcecount =  scalar(@sourceids)."\n";



# This section spawns the initial 4 children.

for ($counter = 1; $counter <= $NumChilds; $counter++) {

#foreach my $source (@sourceids){

        $pid = fork();

        if ($pid) # parent
        {
        $child{$pid} = $counter;
        print "forked process $counter.\n";
        }

        else # child
        {
        callScan();
        #should be callScan($source);
        exit $counter;
        }
#  }
}


#  spawns additional children .

$diecount = 0;

while ($counter <= $sourcecount)
{
        $doneproc = wait();
        $doneval = $? >> 8;
        $pid = fork();

        if ($pid) # parent
        {
        $child{$pid} = $counter;
        print "child $doneval ($doneproc) exited, forking process $cou
+nter.\n";
        $counter++;
        $diecount++;
        }

        else # child
        {
        callScan();
        #should be callScan($source);
        exit $counter;
        }
}


#  waits for all children to die.

while ($diecount <= $sourcecount)
{
        wait();
        $diecount++;
}

# subroutine.
sub callScan
{
        sleep(2);
        print "callScan with counter = $counter.\n";
}
print "Done.\n";

Thanks in advance

[zentara]

Check out Parallel::ForkManager such as controlling child processes

Or do it manually with these old beauties:

#!/usr/bin/perl

=head1
#by Abigail of perlmonks.org
#Some times you have a need to fork of several children, but you want 
+to
#limit the maximum number of children that are alive at one time. Here
#are two little subroutines that might help you, mfork and afork. They
+ are very similar.
#They take three arguments,
#and differ in the first argument. For mfork, the first
#argument is a number, indicating how many children should be forked. 
+For
#afork, the first argument is an array - a child will be
#forked for each array element. The second argument indicates the maxi
+mum
#number of children that may be alive at one time. The third argument 
+is a
#code reference; this is the code that will be executed by the child. 
+One
#argument will be given to this code fragment; for mfork it will be an
+ increasing number,
#starting at one. Each next child gets the next number. For afork, the
+ array element is
#passed. Note that this code will assume no other children will be spa
+wned,
#and that $SIG {CHLD} hasn't been set to IGNORE.
=cut

#mfork example
mfork (10,10,\&hello);
sub hello{print "hello world\n";}
print "all done now\n";


#afork example
if ($#ARGV < 0){@ARGV = qw( 1 2 3 4 5)}
afork (\@ARGV,10,\&hello);
print "Main says: All done now\n";

sub hello{
my $data = $_[0];
print "hello world from $data\n";}

###################################################
sub mfork ($$&) {
my ($count, $max, $code) = @_;
foreach my $c (1 .. $count) {
wait unless $c <= $max;
die "Fork failed: $!\n" unless defined (my $pid = fork);
exit $code -> ($c) unless $pid;
}
1 until -1 == wait;
}
##################################################
sub afork (\@$&) {
my ($data, $max, $code) = @_;
my $c = 0;
foreach my $data (@$data) {
wait unless ++ $c <= $max;
die "Fork failed: $!\n" unless defined (my $pid = fork);
exit $code -> ($data) unless $pid;
}
1 until -1 == wait;
}
#####################################################

[themage]

Hi chanakya,

I would recomend you to take a look at POE::Wheel::Run and the POE_Cookbook/Child_Processes_4 and POE_Cookbook/Child_Processes for simple examples on scripts that do stuff similar to what you pretend.

This examples use the POE::Wheel::Run module, that I successfully used in the past for paralel processing.

They are very simple to use, and give you a great deal of control.

[salva]

use Proc::Queue size => 4, qw(run_back waitpids);

my @pids;
for my $source (@sourceids) {
  push @pids, run_back {
    callScan($source);
  };
}

waitpids(@pids);

[chanakya]

themage,zentara thank you for your suggestions and comments.
zentara the mfork and afork are real gems thank you for sharing.

Thank you

[chanakya]

salva thanks for the suggestion of Proc::Queue. Its simple and very effective. zentara I'm interested to know about afork() routine.
Please walk me through the code. I want to know how the childs are reaped by the parent and how the passed code is executed by the afork() routine



sub afork (\@$&) {
my ($data, $max, $code) = @_;
my $c = 0;
foreach my $data (@$data) {
wait unless ++ $c <= $max;
die "Fork failed: $!\n" unless defined (my $pid = fork);
exit $code -> ($data) unless $pid;
}
1 until -1 == wait;
}