in reply to Re^4: Getting started with MCE (the Many-Core Engine)
in thread Getting started with MCE (the Many-Core Engine)


Some helpful tips for processing a large array.

Spawn workers early before creating or obtaining a large array to be used as input data. Dividing the work equally by the number of workers is not recommended for large data sets. A chunk_size value of 4000 or 8000 is fine for large arrays. It doesn't take much (chunk_size wise) for IPC to not become the bottleneck. Finally, workers persist after processing (re: $mce->process). Thus, shutdown workers when completed. This is done for you when the script terminates if omitted.

#!/usr/bin/perl use strict; use warnings; use MCE; use MCE::Candy; my $volume = 26*26; my $max_workers = 4; my $chunk_size = int $volume / $max_workers / 16; my @results; my $mce = MCE->new( max_workers => $max_workers, chunk_size => $chunk_size, gather => MCE::Candy::out_iter_array(\@results), user_func => sub { my ($mce, $chunk_ref, $chunk_id) = @_; my @output; foreach my $item (@{ $chunk_ref }) { push @output, $item++; } $mce->gather($chunk_id, @output); } )->spawn; $mce->process([ 'aa' .. 'zz' ]); $mce->shutdown; print "$_, " for @results; print scalar @results, "\n";

Regards, Mario