Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re: Multi-thread combining the results together

by marioroy (Vicar)
on Jul 25, 2019 at 11:44 UTC ( #11103361=note: print w/replies, xml ) Need Help??


in reply to Multi-thread combining the results together

Hi Marshall,

Inbound two MCE examples in response to monks Discipulus and 1nickt suggesting MCE. The 1st example stores to a hash.

use strict; use warnings; use MCE::Loop; use Data::Dumper; my @tokens = qw( ab bc cd de ef fg gh ); sub build_regex { my ($token) = @_; chop $token; } MCE::Loop->init( chunk_size => 1, max_workers => 4, ); my %result = mce_loop { my $token = $_; my $regex = build_regex($token); my @line_results = grep { $_ ne $token and /$regex/ } @tokens; MCE->gather( $token => \@line_results ) if @line_results; } @tokens; MCE::Loop->finish; print Dumper(\%result), "\n"; __END__ $VAR1 = { 'ef' => [ 'fg' ], 'fg' => [ 'gh' ], 'bc' => [ 'cd' ], 'ab' => [ 'bc' ], 'cd' => [ 'de' ], 'de' => [ 'ef' ] };

The 2nd example stores to an array and maintains order, possible with MCE::Relay which runs orderly.

use strict; use warnings; use MCE::Loop; use Data::Dumper; my @tokens = qw( ab bc cd de ef fg gh ); sub build_regex { my ($token) = @_; chop $token; } MCE::Loop->init( chunk_size => 1, max_workers => 4, init_relay => 1, ); my @result = mce_loop { my $token = $_; my $regex = build_regex($token); my @line_results = grep { $_ ne $token and /$regex/ } @tokens; MCE::relay { MCE->gather(\@line_results) if @line_results; }; } @tokens; MCE::Loop->finish; print Dumper(\@result), "\n"; __END__ $VAR1 = [ [ 'bc' ], [ 'cd' ], [ 'de' ], [ 'ef' ], [ 'fg' ], [ 'gh' ] ];

Regards, Mario

Replies are listed 'Best First'.
Re^2: Multi-thread combining the results together
by marioroy (Vicar) on Jul 25, 2019 at 12:18 UTC

    Sharing data is feasible using a parallel module of your choice. Here is a demonstration using Parallel::ForkManager and MCE::Shared.

    use strict; use warnings; use Parallel::ForkManager; use MCE::Shared; use Data::Dumper; my @tokens = qw( ab bc cd de ef fg gh ); sub build_regex { my ($token) = @_; chop $token; } my $pm = Parallel::ForkManager->new(4); $pm->set_waitpid_blocking_sleep(0); tie my %result, 'MCE::Shared'; DATA_LOOP: foreach my $token ( @tokens ) { # forks and returns the pid for the child my $pid = $pm->start and next DATA_LOOP; my $regex = build_regex($token); my @line_results = grep { $_ ne $token and /$regex/ } @tokens; $result{ $token } = \@line_results if @line_results; $pm->finish(0); } $pm->wait_all_children; print Dumper( tied(%result)->export({ unbless => 1 }) ), "\n"; __END__ $VAR1 = { 'ef' => [ 'fg' ], 'fg' => [ 'gh' ], 'cd' => [ 'de' ], 'de' => [ 'ef' ], 'bc' => [ 'cd' ], 'ab' => [ 'bc' ] };

    Regards, Mario

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11103361]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (5)
As of 2019-11-16 01:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Strict and warnings: which comes first?



    Results (85 votes). Check out past polls.

    Notices?