Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: Parallel running the process

by marioroy (Sexton)
on Nov 25, 2012 at 06:59 UTC ( #1005467=note: print w/ replies, xml ) Need Help??


in reply to Parallel running the process

MCE is a new Perl module recently added to CPAN. This is how one may use MCE to process in Parallel. MCE is both a chunking and parallel engine. In this case, chunk_size is set to 1. That option is not needed as calling the foreach method will set it to 1 anyway.

The sendto method can be used to serialize data from workers to a file. MCE also provides a do method as well to pass data to a callback function which runs from the main process.

$chunk_ref is a reference to an array. MCE provides both foreach and forchunk methods. In this case, the array contains only 1 entry due to chunk_size being set to 1.

The main page at http://code.google.com/p/many-core-engine-perl/ contains three images. The 2nd one shows the bank queuing model used in MCE with chunking applied to it.

use MCE; ## Parse command line argument for $database_list my $mce = MCE->new( max_workers => 4, chunk_size => 1 ); $mce->foreach("$database_list", sub { my ($self, $chunk_ref, $chunk_id) = @_; my $database = $chunk_ref->[0]; my @result = (); ## Query the database $self->sendto('file:/path/to/result.out', @result); });


Comment on Re: Parallel running the process
Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1005467]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (11)
As of 2014-12-26 09:34 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (171 votes), past polls