Re^2: Parallelization of multiple nested loops

[marioroy]

Hi, cmk

Very cool! Seeing my old parallel code made me want to try again for a 100% pure Perl solution not involving Inline::C. So first serial and parallel code afterwards. Workers consume about 16 MB each. The parallel code runs 3 times faster compared to serial. This is made possible by having workers write to STDOUT directly.

$ time perl serial.pl >/dev/null
$ time perl parallel.pl >/dev/null

Serial Demonstration

use strict;
use warnings;

my @vals = qw( 0.0 0.2 0.4 0.6 0.8 1.0 );

sub proc {
  for my $a ( @vals ) {
  for my $b ( @vals ) {
  for my $c ( @vals ) {
  for my $d ( @vals ) {
  for my $e ( @vals ) {
  for my $f ( @vals ) {
  for my $g ( @vals ) {
  for my $h ( @vals ) {
  for my $i ( @vals ) {
  for my $j ( @vals ) {
  for my $k ( @vals ) {
    print "$a\t$b\t$c\t$d\t$e\t$f\t$g\t$h\t$i\t$j\t$k\t1\t1\n";
  }}}}}}}}}}}
}

proc();

Parallel Demonstration

use strict;
use warnings;
use MCE;

my @vals = qw( 0.0 0.2 0.4 0.6 0.8 1.0 );

# Must autoflush because workers write to STDOUT directly.
STDOUT->autoflush(1);

sub proc {
  my $mce = MCE->new(
    max_workers => scalar(@vals),
    chunk_size  => 1,
    init_relay  => 1,
    user_func   => sub {
      my ($a, $b, $c) = @{ MCE->user_args };
      my ($buf, $d  ) = ( '', $_ );  # $d is the input

      for my $e ( @vals ) {
      for my $f ( @vals ) {
      for my $g ( @vals ) {
      for my $h ( @vals ) {
      for my $i ( @vals ) {
      for my $j ( @vals ) {
      for my $k ( @vals ) {
        $buf .= "$a\t$b\t$c\t$d\t$e\t$f\t$g\t$h\t$i\t$j\t$k\t1\t1\n";
      }}}}}}}

      # Relay is driven by the chunk_id value behind the scene.
      # The benefit is orderly output, one worker at a time.
      MCE::relay { print $buf };
    }
  )->spawn;

  for my $a ( @vals ) {
    for my $b ( @vals ) {
      for my $c ( @vals ) {
        # MCE workers persist between each run. The user_args option
        # is how to pass parameters to them.
        $mce->process({
          input_data => \@vals,
          user_args  => [ $a, $b, $c ],
        });
      }
    }
  }

  $mce->shutdown;
}

proc();

Parallel happens at the 4th level to minimize memory consumption.

Regards, Mario