Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Help with multiple forks

by Eliya (Vicar)
on May 30, 2012 at 16:37 UTC ( #973336=note: print w/ replies, xml ) Need Help??


in reply to Help with multiple forks

Getting back to Parallel::ForkManager, which you attempted to use in your first post, you might want to try

use Parallel::ForkManager; my $fm1 = new Parallel::ForkManager(5); for my $val1 (1..10) { $fm1->start and next; # $0 = "processing step 1: $val1"; sleep 10; open FILE, ">", "$val1.txt" or die $!; print FILE "Output of step 1\n"; close FILE; my $fm2 = new Parallel::ForkManager(2); for my $val2 (qw/a b/) { $fm2->start and next; # $0 = "processing step 2: $val1/$val2"; sleep 5; open FILE1, "<", "$val1.txt" or die $!; open FILE2, ">", "$val2$val1.txt" or die $!; while (my $line = <FILE1>) { $line =~ s/1/2/; print FILE2 $line . "\n"; } close FILE1; close FILE2; $fm2->finish; } $fm1->finish; }

This would run max 5 processes at the outer level, plus max 2*5 at the inner level, i.e. max 15 processes total (10 inner from the last round + 5 outer for the next round).  I'm not entirely sure that's what you want, but it's at least something to play with...

P.S. If you uncomment the $0 = ... lines, you can grep for "processing" in the ps output to observe what's going on.


Comment on Re: Help with multiple forks
Select or Download Code
Re^2: Help with multiple forks
by Anonymous Monk on May 31, 2012 at 09:33 UTC

    Thanks a lot! That's exactly what I was trying to write. I'm a little ashamed because it seems I should have been able to come up with it myself.

    There is one funny thing, though: If I comment out the "for my $val2 (@array2) {..." and the corresponding closing curly bracket (and change $val2 to a constant like "step2"), it breaks with Cannot start another process while you are in the child process at .../perl/lib/perl5/Parallel/ForkManager.pm line 463. Not that I would want that - in my case I need the two nested loops. It's just that I don't know what is happening there. I mean, in the nested loop, I would have assumed I was in the 1st child process, which is itself the parent of the 2nd child process, right?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://973336]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (12)
As of 2015-07-02 10:04 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (33 votes), past polls