Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re: Hobo with a bit of recursion

by marioroy (Vicar)
on Jun 20, 2018 at 23:00 UTC ( #1217082=note: print w/replies, xml ) Need Help??


in reply to Hobo with a bit of recursion

Greetings, Veltro,

I came back to revisit this and did a test. There is a bug in MCE::Shared. The shared hash via the TIE interface and specifying the module option, set to 'MCE::Shared::Hash', should deeply-share automatically when passing key-value pairs during construction.

Thank you, for posting. I will make a new release v1.837 with the fix.

Test Script

use strict ; use warnings ; use feature 'say' ; use Clone 'clone' ; use MCE::Hobo ; use MCE::Shared ; use Data::Dumper ; my $data = { key => 1, nested => { key => 1 } } ; tie my %hash_a, 'MCE::Shared', { module => 'MCE::Shared::Hash' }, %{ c +lone($data) } ; tie my %hash_b, 'MCE::Shared', { module => 'MCE::Shared::Hash' } ; tie my %hash_c, 'MCE::Shared', %{ clone($data) } ; # defaults to MCE: +:Shared::Hash tie my %hash_d, 'MCE::Shared' ; %hash_b = %{ clone($data) } ; %hash_d = %{ clone($data) } ; mce_async { $hash_a{key}++ ; $hash_a{nested}{key}++ ; $hash_b{key}++ ; $hash_b{nested}{key}++ ; $hash_c{key}++ ; $hash_c{nested}{key}++ ; $hash_d{key}++ ; $hash_d{nested}{key}++ ; my $_a = $hash_a{nested} ; $_a->{key}++ ; my $_b = $hash_b{nested} ; $_b->{key}++ ; my $_c = $hash_c{nested} ; $_c->{key}++ ; my $_d = $hash_d{nested} ; $_d->{key}++ ; say "ref nested_a: ", ref($_a) ; say "ref nested_b: ", ref($_b) ; say "ref nested_c: ", ref($_c) ; say "ref nested_d: ", ref($_d) ; say "" ; } ; MCE::Hobo->waitall ; say "a: ", Dumper( tied(%hash_a)->export ) ; # not deeply shared say "b: ", Dumper( tied(%hash_b)->export ) ; # ok say "c: ", Dumper( tied(%hash_c)->export ) ; # ok say "d: ", Dumper( tied(%hash_d)->export ) ; # ok

Output

ref nested_a: HASH ref nested_b: MCE::Shared::Object ref nested_c: MCE::Shared::Object ref nested_d: MCE::Shared::Object a: $VAR1 = bless( { 'key' => 2, 'nested' => { 'key' => 1 } }, 'MCE::Shared::Hash' ); b: $VAR1 = bless( { 'key' => 2, 'nested' => bless( { 'key' => 3 }, 'MCE::Shared::Hash' ) }, 'MCE::Shared::Hash' ); c: $VAR1 = bless( { 'key' => 2, 'nested' => bless( { 'key' => 3 }, 'MCE::Shared::Hash' ) }, 'MCE::Shared::Hash' ); d: $VAR1 = bless( { 'key' => 2, 'nested' => bless( { 'key' => 3 }, 'MCE::Shared::Hash' ) }, 'MCE::Shared::Hash' );

Changes

In the meantime, here are the changes needed in your script to run properly (3 places). The ref is a 'MCE::Shared::Object'.

47,48c47,48 < tie %{$test}, 'MCE::Shared', { module => 'MCE::Shared::Hash' }, %{$_ +test} ; < print Dumper( $test ) ; --- > tie %{$test}, 'MCE::Shared', %{$_test} ; > print Dumper( tied(%{$test})->export ) ; 54c54 < if ( ref $in->{ $_ } eq 'HASH' ) { --- > if ( ref $in->{ $_ } ) { 79c79 < print Dumper( $test ) ; --- > print Dumper( tied(%{$test})->export ) ;

Regards, Mario

Replies are listed 'Best First'.
Re^2: Hobo with a bit of recursion
by Veltro (Friar) on Jun 21, 2018 at 21:03 UTC

    Hello marioroy,

    Thanks for the warning. Luckily I didn't run into this problem since I started to build on one of your examples using the shorthand tie my %test, 'MCE::Shared' ;

    The sharing mechanism actually works all very well using the above statement causing me quite some problems because I did something rather dull. In this example below I successfully managed to destroyed the original hash $h because I didn't realize the consequence of the nested reference (LOL):

    use strict ; use warnings ; use MCE::Shared ; use Data::Dumper ; my $h = { L1_counter => 1, nested1 => { L2_counter => 1, nested2 => { L3_counter => 1, }, }, } ; my @dupkeys = qw{ nested1 } ; tie my %result, 'MCE::Shared' ; @result{@dupkeys} = @{$h}{@dupkeys} ; # Whoops! print Dumper( $h ) ; __END__ $VAR1 = { 'nested1' => { 'L2_counter' => 1, 'nested2' => bless( [ 3, 'MCE::Shared::Hash' ], 'MCE::Shared::Object' ) }, 'L1_counter' => 1 };

    Don't worry too much about whatever I am trying. I am still in the process of understanding the MCE library and for me this is all a big learning exercise for personal development/hobby and all your help is greatly appreciated.

    Couple of other things that I ran into during my endeavors are MCE::Hobo->pending();. It returns 0 when called inside a async block. Is there another way to get the number of pending threads?

    Another thing I was wondering about, if it is possible to create an hash/object and transfer the control to the mother process (without using the sharing mechanism). This would be useful if the object can be created autonomously by a worker process and cut out the extra overhead needed for sharing. Especially if this could be done by passing a reference. (Maybe $mce->gather)?

    Thanks,
    Veltro

      Update: Added demonstration using MCE::Inbox

      Greetings Veltro,

      * I successfully managed to destroyed the original hash $h because I didn't realize the consequence of the nested reference...

      Nested structure is moved to the shared manager process where the shared data resides. Use clone to not alter the original hash.

      use Clone 'clone' ; my $h = { ... } ; my @dupkeys = qw{ nested1 } ; tie my %result, 'MCE::Shared' ; @result{ @dupkeys } = @{ clone($h) }{ @dupkeys } ; print Dumper( $h ) ;


      * MCE::Hobo->pending returns 0 when called inside a async block. Is there another way to get the number of pending threads?

      MCE::Hobo supports nested spawning. The worker hasn't spawned any Hobo's. A way is via messaging between workers and the manager process. See example on Github using MCE::Inbox (not yet released on MetaCPAN).

      use strict; use warnings; use MCE::Hobo; use MCE::Inbox; ## # MCE::Inbox is not yet released on MetaCPAN. It can be used with any # parallel module of choice to include threads. # # Specify mailbox names during construction. Behind the scene, # makes a MCE::Shared->queue object per mailbox. ## my $inbox = MCE::Inbox->new('mngrbox', 'donebox'); MCE::Hobo->init( posix_exit => 1 ); # Workers mce_async { sleep 5; $inbox->send('mngrbox', { ident => 'id1', key => 'value1' }); # do other work, perhaps more send('mngrbox', ...) $inbox->send('donebox', 1); }; mce_async { sleep 1; $inbox->send('mngrbox', { ident => 'id2', key => 'value2' }); # do other work, perhaps another send('mngrbox', ...) $inbox->send('donebox', 1); }; mce_async { $inbox->send('mngrbox', { ident => 'id3', key => 'value3' }); # do other work, perhaps send('mngrbox', ...) in a loop $inbox->send('donebox', 1); }; # Manager while ( my $ret = $inbox->recv('mngrbox') ) { printf "ident: %s, key: %s\n", $ret->{ident}, $ret->{key}; MCE::Hobo->waitone if $inbox->recv_nb('donebox'); # non blocking last unless MCE::Hobo->pending; }


      * Another thing I was wondering about, if it is possible to create an hash/object and transfer the control to the mother process (without using the sharing mechanism).

      The join method in MCE::Hobo is wantarray-aware. It is useful for obtaining data from the hobo process. The data structure is serialized automatically in memory via MCE::Shared.

      use strict; use warnings; use MCE::Hobo; MCE::Hobo->init( posix_exit => 1 ); my @hobos; push @hobos, mce_async { sleep 5; return { ident => 1, key => 'value1' }; }; push @hobos, MCE::Hobo->create( sub { sleep 1; return { ident => 2, key => 'value2' }; }); push @hobos, MCE::Hobo->create( sub { my ($ident) = @_; return { ident => $ident, key => 'value3' }; }, 3 ); # MCE::Hobo->waitall; while ( MCE::Hobo->pending ) { my $ret = MCE::Hobo->waitone->join; print "ident: ", $ret->{ident}, ", key: ", $ret->{key}, "\n"; }

      Another way is the on_finish callback which runs under the mother process. It resembles the on_finish callback in Parallel::ForkManager.

      use strict; use warnings; use MCE::Hobo; MCE::Hobo->init( posix_exit => 1, on_finish => sub { my ( $pid, $exit, $ident, $signal, $error, $ret ) = @_; print "ident: $ident, key: ", $ret->{key}, "\n"; } ); MCE::Hobo->create('id1', sub { sleep 5; return { key => 'value1' }; }); MCE::Hobo->create('id2', sub { sleep 1; return { key => 'value2' }; }); MCE::Hobo->create('id3', sub { return { key => 'value3' }; }); # MCE::Hobo->waitall; while ( MCE::Hobo->pending ) { MCE::Hobo->waitone; }

      Regards, Mario

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1217082]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (8)
As of 2019-11-20 12:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Strict and warnings: which comes first?



    Results (97 votes). Check out past polls.

    Notices?