Parallel::ForkManager + MCE::Shared demonstration

[marioroy]

Respected Monks,

In preparation for the upcoming MCE 1.828 and MCE::Shared 1.825 releases, am testing various modules with MCE::Shared. One of which is Parallel::ForkManager.

Discipulus introduced me to zentara recently. zentara am pleased to meet you. Discipulus, imho, folks may choose any parallel module of their liking. It doesn't need to be MCE and the reason for this thread. I like Parallel::ForkManager too.

Some time back, zentara wrote a Parallel::ForkManager + IPC::Shareable demonstration. Fast forward 2.5 years and here's another way. MCE::Shared provides users of Parallel::ForkManager with threads-like sharing capabilities. Below is zentara's example updated with MCE::Shared bits.

Not to worry, MCE::Shared performs reasonably well.

#!/usr/bin/perl

# Based on Parallel::ForkManager + IPC::Shareable by zentara.
# Found here: http://www.perlmonks.org/?node_id=1104697

use strict;
use Parallel::ForkManager;
use MCE::Mutex;
use MCE::Shared;

my $mutex = MCE::Mutex->new();
my $parent_share = tie my %final_parent_hash, 'MCE::Shared';

my $fork_manager = new Parallel::ForkManager(5);
   $fork_manager->set_waitpid_blocking_sleep(0);

foreach my $child ( 1 .. 10 ) {
    my $pid = $fork_manager->start($child) and next;

    # Optional, to have the shared-manager assign a data channel.
    # Helpful when involving heavy IPC usage, not the case here.
    # Increase 20 to 2000 to see perf-increase from calling ->init.
    MCE::Shared->init();

    for my $id ( 1 .. 20 ) {
        my $key = $child . '-' . $id;
      # $mutex->lock; # mutex not necessary when storing unique keys
      # $parent_share->set($key => qq{|Kid $child pushed $id}); # OO
        $final_parent_hash{$key} = qq{|Kid $child pushed $id};
      # $mutex->unlock;
    }

    $fork_manager->finish($child);
}

print "Waiting for Children...\n";
$fork_manager->wait_all_children;

foreach my $child ( 1 .. 10 ) {
    for my $id ( 1 .. 20 ) {
        my $key = $child . '-' . $id;
        if (! exists $final_parent_hash{$key} ) {
            print "Missing data for Kid $child , data $id\n";
        }
        else {
            print "$key = $final_parent_hash{$key}\n";
        }
    }
}

The following are recommended modules for MCE::Shared.

## MCE::Shared

1. Sereal::Decoder 3.015+
2. Sereal::Encoder 3.015+
3. Sereal (ok for completeness, but MCE::Shared doesn't load this)

## MCE::Shared applies to Condvar, Handle, and Queue

1. IO::FDPass 1.2+

Q. Why is Sereal beneficial?

A. The main reason is for extra performance. To ensure minimum memory consumption, there's no reason to load the Storable module if Sereal is available in Perl. This is handled transparently.

Q. Why is IO::FDPass beneficial?

A. Being able to construct a shared condvar, handle, or queue while the shared-manager is running greatly adds to the ease-of-use. These involve handles behind the scene. Basically, am able to send the relevant fd descriptors to the shared-manager. Without FDPass, one must be careful to construct Condvar, Handle, and Queue first before other shared objects and later starting the shared-manager manually. Note: MCE and MCE::Hobo starts the shared-manager if not already started.

Q. What is MCE::Shared->init all about?

A. For MCE, MCE::Hobo, and threads (via CLONE), MCE::Shared->init() is called automatically. It assigns the worker 1 of 12 data channels for use during IPC. Calling init is totally optional. If the worker is sending data one time, probably not necessary. On the other hand, if doing lots of IPC, then yes worth it.

For further reading, see also this thread made by karlgoethebier or this reply regarding performance characteristics (TIE and Mutex or OO). Basically, performance is possible. And so is fun.

Regards, Mario.