|Just another Perl shrine|
Parallelization of multiple nested loopsby biosub (Initiate)
|on Feb 07, 2018 at 09:44 UTC||Need Help??|
biosub has asked for the
wisdom of the Perl Monks concerning the following question:
I'm a bit of a newbie in Perl, I'm a biologist who codes for bioinformatics and I don't come across Perl that often. But here I am..
I'm trying to implement parallelization in someone else code that searches through a very big parameter space (a.k.a. all the possible combinations of the possible values of the defined parameters).
I managed to use Parallel:ForkManager for the rest of the code but I'm stuck with these nested loops. Here's the code:
The problem is that with the forking inside the last loop it only starts max 6 processes, because with the current settings each loop runs 6 times. Although being an improvement this is not what I aim for.
I tried putting the forking higher in the loop structure: it does run more procesess but it only returns the last combination of the loop.
I tried nested forking as suggested here https://www.perlmonks.org/?node_id=973304 (the commented out code with $fm_outer_1 variable) but it only issues and endless repetition of the warning:
and a few interspersed warnings of children not returining anything to the parent.
I tried setting up data retrival as for the $fork_manager and to shuffle around the positions of the wait_all_children() but without luck.
I resort to the Monks for help in either write a proper parallelization (I'm running on a 64 core server, so I'd like to use the power that's there ;) ) or, if exists, to point me to some package/module that can do the same thing in a bit more clean way that writing a bunch of nested loops.I hope everything is clear!