Re: Help with multiple forks

[Eliya]

in reply to Help with multiple forks

Getting back to Parallel::ForkManager, which you attempted to use in your first post, you might want to try

use Parallel::ForkManager;

my $fm1 = new Parallel::ForkManager(5);

for my $val1 (1..10) {
    
    $fm1->start and next;

    # $0 = "processing step 1: $val1";  sleep 10;
    
    open FILE, ">", "$val1.txt" or die $!;
    print FILE "Output of step 1\n";
    close FILE;

    my $fm2 = new Parallel::ForkManager(2);
    
    for my $val2 (qw/a b/) {

        $fm2->start and next;

        # $0 = "processing step 2: $val1/$val2";  sleep 5;

        open FILE1, "<", "$val1.txt" or die $!;
        open FILE2, ">", "$val2$val1.txt" or die $!;
        while (my $line = <FILE1>) {
            $line =~ s/1/2/;
            print FILE2 $line . "\n";
        }
        close FILE1;
        close FILE2;
        
        $fm2->finish;
    }
    
    $fm1->finish;
}

This would run max 5 processes at the outer level, plus max 2*5 at the inner level, i.e. max 15 processes total (10 inner from the last round + 5 outer for the next round).  I'm not entirely sure that's what you want, but it's at least something to play with...

P.S. If you uncomment the $0 = ... lines, you can grep for "processing" in the ps output to observe what's going on.

[Anonymous Monk]

Thanks a lot! That's exactly what I was trying to write. I'm a little ashamed because it seems I should have been able to come up with it myself.

There is one funny thing, though: If I comment out the "for my $val2 (@array2) {..." and the corresponding closing curly bracket (and change $val2 to a constant like "step2"), it breaks with Cannot start another process while you are in the child process at .../perl/lib/perl5/Parallel/ForkManager.pm line 463. Not that I would want that - in my case I need the two nested loops. It's just that I don't know what is happening there. I mean, in the nested loop, I would have assumed I was in the 1st child process, which is itself the parent of the 2nd child process, right?