Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

New stable MCE 1.842 and MCE::Shared 1.842 releases

by marioroy (Vicar)
on Jul 22, 2019 at 03:13 UTC ( #11103120=CUFP: print w/replies, xml ) Need Help??

Update: Added results for Parallel::ForkManager 2.02. The results mentioned in the POD documentation ran slower due to an unaware background job at the time. I reran again on all 5 platforms.

Dear fellow Monks,

I am pleased to annouce MCE 1.842 and MCE::Shared 1.842 (both stable). MCE now includes MCE::Channel and MCE::Child recently.

This weekend, added Parallel::ForkManager-like demonstration to the POD section in MCE::Child and MCE::Hobo. The results were captured on a Macbook Pro (late 2013 model - 2.6 GHz ~ 3.6 GHz with Turbo Boost).

To run, one may direct standard output to :nul or /dev/null depending on the platform. Or better yet, I commented out the print line in the on_finish handler. Unfortunately, Parallel::ForkManager 2.02 suffers from memory leaks on the Windows platform ($^O eq 'MSWin32') and the reason why slower than Cygwin.

Time to spin 2,000 workers and obtain results (in seconds).

MCE::Hobo uses MCE::Shared to retrieve data during reaping. MCE::Child uses MCE::Channel, no shared-manager. Version Cygwin Windows Linux macOS FreeBSD MCE::Child 1.842 19.099s 17.091s 0.965s 1.534s 1.229s MCE::Hobo 1.842 20.514s 19.594s 1.246s 1.629s 1.613s P::FM 1.20 19.703s 19.235s 0.875s 1.445s 1.346s MCE::Child 1.842 20.426s 18.417s 1.116s 1.632s 1.338s Moo loaded MCE::Hobo 1.842 21.809s 20.810s 1.407s 1.759s 1.722s Moo loaded P::FM 2.02 21.668s 25.927s 1.882s 2.612s 2.483s Moo used

Set posix_exit to avoid all END and destructor processing.

MCE::Child->init( posix_exit => 1, ... ); MCE::Hobo->init( posix_exit => 1, ... ); Version Cygwin Windows Linux macOS FreeBSD MCE::Child 1.842 19.815s ignored 0.824s 1.284s 1.245s Moo loaded MCE::Hobo 1.842 21.029s ignored 0.953s 1.335s 1.439s Moo loaded


use strict; use warnings; use Parallel::ForkManager; use Time::HiRes 'time'; my $start = time; my $pm = Parallel::ForkManager->new(10); $pm->set_waitpid_blocking_sleep(0); $pm->run_on_finish( sub { my ($pid, $exit_code, $ident, $exit_signal, $core_dumped, $resp) = + @_; # print "child $pid completed: $ident => ", $resp->[0], "\n"; }); DATA_LOOP: foreach my $data ( 1..2000 ) { # forks and returns the pid for the child my $pid = $pm->start($data) and next DATA_LOOP; my $ret = [ $data * 2 ]; $pm->finish(0, $ret); } $pm->wait_all_children; printf STDERR "duration: %0.03f seconds\n", time - $start;


use strict; use warnings; use MCE::Child 1.842; use Time::HiRes 'time'; my $start = time; MCE::Child->init( max_workers => 10, on_finish => sub { my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @ +_; # print "child $pid completed: $ident => ", $resp->[0], "\n"; } ); foreach my $data ( 1..2000 ) { MCE::Child->create( $data, sub { [ $data * 2 ]; }); } MCE::Child->wait_all; printf STDERR "duration: %0.03f seconds\n", time - $start;


use strict; use warnings; use MCE::Hobo 1.842; use Time::HiRes 'time'; my $start = time; MCE::Hobo->init( max_workers => 10, on_finish => sub { my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @ +_; # print "child $pid completed: $ident => ", $resp->[0], "\n"; } ); foreach my $data ( 1..2000 ) { MCE::Hobo->create( $data, sub { [ $data * 2 ]; }); } MCE::Hobo->wait_all; printf STDERR "duration: %0.03f seconds\n", time - $start;

These days, there are other ways and instead have workers persist (i.e. pull items from a shared queue or channel) or perhaps via MCE's input and chunking capabilities.

Regards, Mario

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: CUFP [id://11103120]
Approved by marto
Front-paged by marto
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (5)
As of 2020-07-13 20:45 GMT
Find Nodes?
    Voting Booth?

    No recent polls found