how to limit child processes?

[Galen]

I'm using fork() and having trouble limiting the number of child processes that spawn. Say I have 12 tasks to do in a list. My technique is to to set a batch size ($children = 5) and process the list one by one, counting as I go along. If I've reached the batch size ($counter mod $children becomes false) or I'm at the end of the list, then go ahead and execute on this batch. Executing involves forking out $children number of child processes that take 15-20 seconds each to complete.

The problem is that even though the logical order of my code says to: 1) create the batch, then 2) do something with this batch, then 3) create another batch, it isn't working in that order. Instead, I watch as 12 child processes go out and do their thing. Even though part of the duty of each process is to print data and that data has not yet been returned to me, the loop figures it's done and proceeds to the next batch. I don't get it.

Here's an example - shouldn't this snippet send off 5 processes which sleep for 15 seconds, then print 5 values, after which another 15 seconds passes and another 5 values are printed, then 15 seconds and the last two are printed? Instead, it sleeps for 15 seconds and prints all 12 values.

#!/usr/bin/perl
# batch.pl

use IO::Select;
use IO::Pipe;

my @list = ("a", "b", "c", "d", "e", "f", 
            "g", "h", "i", "j", "k", "l");
my $children = 5;
my $counter = 0;
my $size = @list;

foreach (@list) {
  $counter++;
  if (($counter%$children) && ($counter != $size)) {
    push(@temp,$_);
  } else {
    push(@temp,$_);
    @batch = @temp;
    @temp = @blank;
    foreach (@batch) {
      my $pipe = IO::Pipe->new();
      my $pid = fork;
      die "Bad Fork $!" unless defined $pid;

      if ($pid) {
        $pipe->reader();
#        $select->add( $pipe );

      } else {
        $pipe->writer();
        $pipe->autoflush(1);

        &dosomething;
        exit;
      }
    }
  }
}

sub dosomething {
  sleep 15;
  print "$_\n";
}

By the way, I tried using waitpid() with no success. Perhaps I was using it incorrectly. Is waitpid() the way to go, or is my logic just totally wrong?

[AgentM]

waitpid IS the way to go but perhaps the code from Parallel::ForkManager or Fork::Queue can give you a good start.


AgentM Systems nor Nasca Enterprises nor Bone::Easy nor Macperl is responsible for the comments made by AgentM. Remember, you can build any logical system with NOR.

[merlyn]

As is the code and explanation from one of my columns.

-- Randal L. Schwartz, Perl hacker

[mikfire]

Not that my code will be so great after reading merlyn's, but here is my general solution to this problem. Unless you read the rest of the thread, you can likely ignore the first two paragraphs :)

PS - AgentM was dead on with his waitpid comment

mikfire