Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Re: Parallelization of multiple nested loops

by ikegami (Pope)
on Feb 07, 2018 at 23:50 UTC ( #1208662=note: print w/replies, xml ) Need Help??

in reply to Parallelization of multiple nested loops

You are actually making your program slower by forking! You are doing so little in the children that deserializing the results from the child is slower than actually generating them in the first place! You need to do a lot more work per child.

You had the right idea about including more levels of looping in the child.

use Algorithm::Loops qw( NestedLoops ); use Parallel::ForkManager qw( ); my $num_jobs_log6 = 4; # 6*6*6*6 jobs. my $num_workers = 64; # 20.25 jobs/worker (on average). my $pm = Parallel::ForkManager->new($num_workers); my @param_arr; $pm->run_on_finish(sub{ my ($pid, $exit_code, $ident, $exit_signal, undef, $data) = @_; die("Child killed by signal $exit_signal\n") if $exit_signal; die("Child return error $exit_code\n") if $exit_code; push @param_arr, @$data; }); my @ps = map { $_ * 0.2 } 0..5; my $outer_iter = NestedLoops([ (\@ps) * $num_jobs_log6 ]); while (my @p_outer = $outer_iter->()) { my $p_outer = join("\t", @p_outer); $pm->start() and next; my @child_output; my $inner_iter = NestedLoops([ (\@ps) * ( 11 - $num_jobs_log6 ), [ 1 ], [ 1 ], ]); while (my @p_inner = $inner_iter->()) { push @child_output, $p_outer . join("\t", @p_inner) . "\n"; } $pm_finish(0, \@output); } $pm->wait_all_children();

I'm still not convinced that this will be faster than doing it all in one process. You do more work in the child, but you also increase the amount of time needed to deserialize the result of the child.

I think you are optimizing the wrong part of your code! If this part of the code is really slowing you down, just pre-generate the values in a file and load that file in the array.

my @param_arr; { open(my $fh, '<', $qfn) or die("Can't open \"$qfn\": $!\n"); @param_arr = <$fh>; }

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1208662]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (5)
As of 2018-05-26 10:42 GMT
Find Nodes?
    Voting Booth?