Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re^2: Camel vs. Gopher

by marioroy (Vicar)
on Dec 13, 2018 at 01:25 UTC ( #1227193=note: print w/replies, xml ) Need Help??


in reply to Re: Camel vs. Gopher
in thread Camel vs. Gopher

Go is cool. So is Perl. :) Below, demonstrations based on Your_Mother's example.

Serial Code

use strict; use warnings; use Time::HiRes 'time'; my $start = time; my @ten = ( 0 .. 9 ); my %count; $count{ $ten[ rand @ten ] }++ for 1..10_000_000; print "$_ -> $count{$_}\n" for @ten; printf "duration: %0.3f seconds\n", time - $start;

Parallel Implementation

Disclaimer: I typically use the OO interface for shared variables. The reason is exclusive locking handled automatically.

use strict; use warnings; use MCE; use MCE::Shared; use Time::HiRes 'time'; my $start = time; my @ten = ( 0 .. 9 ); my $count = MCE::Shared->hash(); MCE->new( max_workers => 4, sequence => [ 1, 10_000_000 ], bounds_only => 1, chunk_size => 50_000, user_func => sub { my ( $mce, $seq, $chunk_id ) = @_; my %lcount; # compute using a local hash - involves zero IPC $lcount{ $ten[ rand @ten ] }++ for $seq->[0] .. $seq->[1]; # increment shared hash - one IPC per key while ( my ( $key, $val ) = each %lcount ) { $count->incrby( $key, $val ); } } )->run; printf "$_ -> %ld\n", $count->get($_) for @ten; printf "duration: %0.3f seconds\n", time - $start;

Results running Perl v5.28, parallel time includes workers spawning-shutdown

Serial 1.534 seconds Parallel 0.410 seconds 0 -> 999455 1 -> 1000312 2 -> 999828 3 -> 1001949 4 -> 999227 5 -> 997375 6 -> 1001048 7 -> 999806 8 -> 1000212 9 -> 1000788

Regards, Mario

Replies are listed 'Best First'.
Re^3: Camel vs. Gopher
by marioroy (Vicar) on Dec 13, 2018 at 01:37 UTC

    The pipeline method batches multiple operations into a single IPC call. A smaller chunk_size value is needed to notice the difference.

    # increment shared hash - single IPC per chunk $count->pipeline( map { [ 'incrby', $_ => $lcount{$_} ] } keys %lcount );

    Comparison chunk_size => 1_000

    Before 0.814 seconds $count->incrby(...) After 0.522 seconds $count->pipeline(...)

    Regards, Mario

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1227193]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (6)
As of 2019-10-18 10:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?