Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Parallel::ForkManager + MCE::Shared demonstration

by marioroy (Priest)
on Apr 24, 2017 at 02:26 UTC ( #1188731=CUFP: print w/replies, xml ) Need Help??

Respected Monks,

In preparation for the upcoming MCE 1.828 and MCE::Shared 1.825 releases, am testing various modules with MCE::Shared. One of which is Parallel::ForkManager.

Discipulus introduced me to zentara recently. zentara am pleased to meet you. Discipulus, imho, folks may choose any parallel module of their liking. It doesn't need to be MCE and the reason for this thread. I like Parallel::ForkManager too.

Some time back, zentara wrote a Parallel::ForkManager + IPC::Shareable demonstration. Fast forward 2.5 years and here's another way. MCE::Shared provides users of Parallel::ForkManager with threads-like sharing capabilities. Below is zentara's example updated with MCE::Shared bits.

Not to worry, MCE::Shared performs reasonably well.

#!/usr/bin/perl # Based on Parallel::ForkManager + IPC::Shareable by zentara. # Found here: http://www.perlmonks.org/?node_id=1104697 use strict; use Parallel::ForkManager; use MCE::Mutex; use MCE::Shared; my $mutex = MCE::Mutex->new(); my $parent_share = tie my %final_parent_hash, 'MCE::Shared'; my $fork_manager = new Parallel::ForkManager(5); $fork_manager->set_waitpid_blocking_sleep(0); foreach my $child ( 1 .. 10 ) { my $pid = $fork_manager->start($child) and next; # Optional, to have the shared-manager assign a data channel. # Helpful when involving heavy IPC usage, not the case here. # Increase 20 to 2000 to see perf-increase from calling ->init. MCE::Shared->init(); for my $id ( 1 .. 20 ) { my $key = $child . '-' . $id; # $mutex->lock; # mutex not necessary when storing unique keys # $parent_share->set($key => qq{|Kid $child pushed $id}); # OO $final_parent_hash{$key} = qq{|Kid $child pushed $id}; # $mutex->unlock; } $fork_manager->finish($child); } print "Waiting for Children...\n"; $fork_manager->wait_all_children; foreach my $child ( 1 .. 10 ) { for my $id ( 1 .. 20 ) { my $key = $child . '-' . $id; if (! exists $final_parent_hash{$key} ) { print "Missing data for Kid $child , data $id\n"; } else { print "$key = $final_parent_hash{$key}\n"; } } }

The following are recommended modules for MCE::Shared.

## MCE::Shared 1. Sereal::Decoder 3.015+ 2. Sereal::Encoder 3.015+ 3. Sereal (ok for completeness, but MCE::Shared doesn't load this) ## MCE::Shared applies to Condvar, Handle, and Queue 1. IO::FDPass 1.2+

Q. Why is Sereal beneficial?

A. The main reason is for extra performance. To ensure minimum memory consumption, there's no reason to load the Storable module if Sereal is available in Perl. This is handled transparently.

Q. Why is IO::FDPass beneficial?

A. Being able to construct a shared condvar, handle, or queue while the shared-manager is running greatly adds to the ease-of-use. These involve handles behind the scene. Basically, am able to send the relevant fd descriptors to the shared-manager. Without FDPass, one must be careful to construct Condvar, Handle, and Queue first before other shared objects and later starting the shared-manager manually. Note: MCE and MCE::Hobo starts the shared-manager if not already started.

Q. What is MCE::Shared->init all about?

A. For MCE, MCE::Hobo, and threads (via CLONE), MCE::Shared->init() is called automatically. It assigns the worker 1 of 12 data channels for use during IPC. Calling init is totally optional. If the worker is sending data one time, probably not necessary. On the other hand, if doing lots of IPC, then yes worth it.

For further reading, see also this thread made by karlgoethebier or this reply regarding performance characteristics (TIE and Mutex or OO). Basically, performance is possible. And so is fun.

Regards, Mario.

Replies are listed 'Best First'.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: CUFP [id://1188731]
Approved by Athanasius
Front-paged by Arunbear
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (2)
As of 2017-11-18 05:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In order to be able to say "I know Perl", you must have:













    Results (277 votes). Check out past polls.

    Notices?