Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much

Getting started with MCE (the Many-Core Engine)

by Anonymous Monk
on Jun 11, 2018 at 06:05 UTC ( #1216368=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to get Perl to use all CPU cores with MCE. I have read and tried some examples but can't seem to grasp it. Can anyone explain MCE for dummies? I was hoping this would produce a large number but it only prints 0:
#!/usr/bin/perl use strict; use warnings; use MCE::Loop; our $fu = 0; MCE::Loop::init { max_workers => 8, chunk_size => 1000 }; mce_loop { do_work($_) } 1..10000000; print "Done: $fu\n"; sub do_work { $fu += $_[0] }

Replies are listed 'Best First'.
Re: Getting started with MCE (the Many-Core Engine)
by vr (Deacon) on Jun 11, 2018 at 08:49 UTC
    use strict; use warnings; use MCE; use MCE::Loop; use MCE::Shared; my $volume = 1e9; # crank it up my $max_workers = MCE::Util::get_ncpu; my $chunk_size = int $volume / $max_workers; MCE::Loop::init { max_workers => $max_workers, chunk_size => $chunk_size, bounds_only => 1, }; my $fu = MCE::Shared-> scalar( 0 ); mce_loop_s { # note "s" my $partial_result = 0; my ( $begin, $end ) = @$_; $partial_result += $_ for $begin .. $end; print 'Worker', MCE-> wid, ' here, my partial result is ', $partial_result, "\n"; $fu-> incrby( $partial_result ); } 1, $volume; print "\nDone: @{[ $fu-> get ]}\n"; __END__ Worker4 here, my partial result is 218750000125000000 Worker1 here, my partial result is 31250000125000000 Worker3 here, my partial result is 156250000125000000 Worker2 here, my partial result is 93750000125000000 Done: 500000000500000000

    Hi, not sure if it explains for dummies or not. Feel free to ask further, if it doesn't. Important:

    • it's required to provide way to communicate results back from workers which are just separate processes (or threads) -- i.e. to arrange for IPC. You didn't, thus your variable remained zero. One possibility, through explicitly shared variable, is shown above;
    • you really don't want to build an enormous list (1e7 elements in your code, or moreover 1e9 above) before any work is even started. For sequences, the MCE provides built-in methods. For other kind of input, think about e.g. setting up iterator as source.

      Eureka! Adding a billion numbers was taking the usual single perl process 26.8 seconds but MCE spawns 8 perls (1 process per CPU core with get_ncpu), giving the fan a nice workout and doing it in 7.3 seconds! 365% faster FTW!
      time perl -le '$x=1e9;$y=0;$y+=$_ for 1..$x;print$y' 500000000500000000 real 0m26.796s user 0m26.617s sys 0m0.088s time mce (your script) 500000000500000000 real 0m7.323s user 0m55.095s sys 0m0.127s
      Thank you so much for the example code. I can now see the meaning of chunk_size and how to share data between processes. MCE is a really beautiful distribution with comprehensive documentation and tons of examples (and a couple of cookbooks). I'm just not familiar with the concepts and jargon of parallel processing and was feeling kinda lost, but less now. Thanks to you! :-)

      Time for another go at the docs while I figure out how to adapt this to incrementing strings perl-style:

      perl -le '$a="a";for(1..30){print$a++}'

        how to adapt this to incrementing strings perl-style

        See a a recent example where I showed exactly that, using tie with MCE::Shared (that example shows incrementing the value of keys in a hash; see the MCE::Shared doc for sharing a scalar via tie).

        Alternatively, if you are using MCE::Shared's OO interface it provides sugar methods (shown here using MCE::Hobo for workers):

        my $shared = MCE::Shared->scalar(0); sub task { $shared->incrby(1); } MCE::Hobo->init( max_workers => 8, posix_exit => 1 ); MCE::Hobo->create( \&task, $_ ) for 0 .. 41; MCE::Hobo->wait_all; END { print "The answer is $shared\n"; }

        Hope this helps!

        The way forward always starts with a minimal test.
        incrementing strings perl-style

        You mean, as in perlop, string

        matches the pattern /^[a-zA-Z]*[0-9]*\z/, the increment is done as a string, preserving each character within its range, with carry
        ? What an unusual and interesting task. What's the context? And string is huge enough for parallelization overhead to pay off? Off the top of my head, split the string into more or less equal substrings (their number equal to number of workers), guaranteed not to carry when incremented required number of times -- or, find indices within original, then feed the data to MCE::Map, then concatenate the output :-)
        Yes I did notice that 8 perl processes vs 1 "only" sped things up about 3.5x. Tests show no gains over 4 workers here. I guess these i7 processors have 4 physical cores virtualized to 8 marketing thingies. Without MCE my computationally intensive code seemed to be using the resources of 1/2 of 1 of the four real cores (1 virtual core? teenage CPU %s). MCE appears to unlock the missing ~85% of my CPU power. It's like I downloaded 3.5 more of these 4k computers from CPAN for FREE with MCE (and 1 post++ from Perlmonk vr): A $14,000 value! Unreal...

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1216368]
Front-paged by Discipulus
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (5)
As of 2019-10-22 21:03 GMT
Find Nodes?
    Voting Booth?