http://www.perlmonks.org?node_id=11130808

exilepanda has asked for the wisdom of the Perl Monks concerning the following question:

I know thread::shared can't share deep data structure, so I attempt to only share the stringify memory address, and deref() it to access the structure inside the child thread. But what I got is really weird. Can someone help to explain how this happen? and any fix for this approach? Thank you very much
$| = 1; use threads; use threads::shared; use Devel::Pointer; use Data::Dumper; $Data::Dumper::Sortkeys; my $Addr :shared; my $Deep = {OK=>1}; $Addr = address_of $Deep; sub A { while ( 1 ) { my $Data = deref $Addr; $Data->{A}++; print "Dumper in A tells:"; print Dumper $Data; print "B is now '$Data->{B}'"; # nothing! $Data->{B} = "Overide by A"; # The output suprised me! print $/; sleep 2; } } sub B { sleep 1; while ( 1 ) { my $Data = deref $Addr; $Data->{B}++; print "Dumper in B tells:"; print Dumper $Data; print "A is now '$Data->{A}'"; print $/; sleep 2; } } my $A = threads -> create ( 'A' ) ; my $B = threads -> create ( 'B' ); $A->detach; $B->detach; while ( 1 ) { $Addr = address_of $Deep ; sleep 1 } __END__ Surprising Output: Dumper in A tells:$VAR1 = { 'A' => 1, 'OK' => 1 }; B is now '' ## Nothing! Dumper in B tells:$VAR1 = { 'A' => 1, 'B' => 1, 'B' => 'Overide by A', ## POSSIBLE?! Same key in Hash?! 'OK' => 1 }; A is now '' ## Nothing too! Dumper in A tells:$VAR1 = { 'A' => 2, 'B' => 1, 'B' => 'Overide by A', 'OK' => 1 }; ...

Replies are listed 'Best First'.
Re: use Devel::Pointer in threads
by LanX (Saint) on Apr 04, 2021 at 15:04 UTC
    Well, I think you are trying to break thread safety and now you're getting bitten by undefined behavior.

    > I know threads::shared can't share deep data structure,

    according to the docs that's not true, you can share deep data structures as long as you also share any nested reference.

    share(VARIABLE)

    Shared variables can only store scalars, refs of shared variables, or refs of shared data (discussed in next section):

    > and any fix for this approach?

    You can always use shared_clone for read access.

    And of course you can explicitly share singular scalars inside your nested data structure to allow writing.

    shared_clone(REF)

    shared_clone takes a reference, and returns a shared version of its argument, performing a deep copy on any non-shared elements. Any shared elements in the argument are used as is (i.e., they are not cloned).

    If that's not good enough because you can't/don't want to predict the structure of your nested data, you might try Data::Dumper or Data::Dump for serializing it into a shared string. The receiving end can easily unpack it again.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

      I have a reason to attempt this, tho I know I will implement my own lock if this approach works. In fact, I have to assume I cannot predict how will the $Deep grows, but the $Deep can be really deep, like JSON. Child threads will read files and then further extend the structure tree, and every other child will work depends on the current tree to dig what they interest. That's why I can't declare explicitly ahead, nor a clone of the current helps.

      I could have a safer approach by using DB, or deploy a thread to implement a TCP socket, so every child talk to the DB/Socket, but I try to use at least of I/O as possible.

        Like I already said, I'd rather serialise data into a long string than to depend on undefined behaviour.

        I mentioned Data::Dump* you mentioned JSON, so what's the problem?

        Your trick to bypass "sharing" can segfault in worst case and is unreliable in best case.

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        Wikisyntax for the Monastery

Re: use Devel::Pointer in threads
by 1nickt (Canon) on Apr 04, 2021 at 14:50 UTC

    Hi, for shared data whether under threads or not you may have more luck with MCE::Shared.

    Hope this helps!


    The way forward always starts with a minimal test.
Re: use Devel::Pointer in threads
by LanX (Saint) on Apr 04, 2021 at 15:48 UTC
    For the why:

    > Can someone help to explain how this happen?

    Perl's model for threading is to clone the interpreter with the data. It's allowing thread-safety by emulating forking.

    The shared data is explicitly tunneled between the threads.

    New data - including their refs - doesn't exist in other threads unless they are shared.

    Hence using a backdoor like Devel::Pointer to the memory address can't work.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

      Thanks but this didn't explain why Data::Dumper can actually fetch what's inside, but I can't when explicitly write to print $Deep->{B}. Also, how a hash can have same key exists when Dump.
Re: use Devel::Pointer in threads
by vr (Curate) on Apr 05, 2021 at 13:07 UTC
    That's why I can't declare explicitly ahead, nor a clone of the current helps

    You don't have to declare ahead, and only have to shared_clone (references to) "composite" fragments (arrays and hashes) which are being appended at each step. Here 3 workers are happily building (somewhat elaborate example, the best I came up with now) shared AoA, where, for simplicity and brevity, appended parts have rigid structure and depth is tracked, but that won't be necessary in practical implementation. Leading digit is creator worker ID, appended digits are IDs of workers which have seen this fragment, and demonstrably were able to modify it. And of course "locking" happens here because of coordinated sleeping pattern, also for simplicity. I hope I understood your requirements right, and building shared common structures in parallel will, indeed, lead to any speed gain.

    use strict; use warnings; $| = 1; use threads; use threads::shared; use Data::Dumper; $Data::Dumper::Indent = 0; my $root = shared_clone [ threads-> tid ]; my $depth: shared = -1; sub worker { my $i = threads-> tid; while ( 1 ) { my $ary = $root; for ( 0 .. $depth ) { $ary-> [0] .= $i; $ary = $ary-> [1] } $ary-> [0] .= $i; $ary-> [1] = shared_clone [ $i ]; $depth ++; print '*'; sleep 3; } } for ( 1 .. 3 ) { threads-> create( \&worker )-> detach; sleep 1 } sleep 2; print "\n", Dumper( $root ), "\n"; __END__ ****** $VAR1 = ['0123123',['123123',['23123',['3123',['123',['23',[3]]]]]]];
Re: use Devel::Pointer in threads
by karlgoethebier (Abbot) on Apr 05, 2021 at 07:32 UTC

    Try something like this. Best regards, Karl

    «The Crux of the Biscuit is the Apostrophe»