Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re: How to share complex data structure in threads ?

by BrowserUk (Pope)
on Dec 30, 2012 at 05:40 UTC ( #1010892=note: print w/ replies, xml ) Need Help??


in reply to How to share complex data structure in threads ?

threads::shared seems only serve for simple array, scalar or hash.

Look again at threads::shared::shared_close(). Eg:

use threads;; use threads::shared; my @r = ( map+{ map{ $_ => [ 1 .. 10 ] } 'a' .. 'z' }, 1..10 );; pp \@r;; [ { a => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], b => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], c => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], d => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], e => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], f => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], g => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], h => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], i => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], j => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], k => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], l => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "m" => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], n => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], o => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], p => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "q" => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], r => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "s" => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], t => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], u => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], v => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], w => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "x" => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "y" => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], z => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], }, { a => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], ... }, ] my @s : shared = map shared_clone( $_ ), @r; pp \@s; [ # tied threads::shared::tie { # tied threads::shared::tie a => [ # tied threads::shared::tie 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ], b => [ # tied threads::shared::tie 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ], c => [ # tied threads::shared::tie 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ], d => [ # tied threads::shared::tie ... ], ... }, { # tied threads::shared::tie a => [ # tied threads::shared::tie 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ], b => ... } ... ]

It won't clone subroutines, because that's not possible or logical.

However, whilst this allows you to share complex data structures, you'll generally find that it doesn't achieve what you think you want to achieve, and you'll probably come back asking why not.

If you would describe your application, there is usually a better way of achieving it.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.


Comment on Re: How to share complex data structure in threads ?
Select or Download Code
Re^2: How to share complex data structure in threads ?
by exilepanda (Monk) on Dec 30, 2012 at 07:11 UTC
    Thanks for the lead. For the sub{} share stuff, I am still not sure it's necessity. But consider the following code, If I give anything try to alter the cloned structure, it give the Thread 2 terminated abnormally: Invalid value for shared scalar error.
    $| = 1; use Data::Dumper; use threads;; use threads::shared; my @r = ( map+{ map{ $_ => [ 1 .. 2 ] } 'a' .. 'b' }, 1..2 );; my @s : shared = map shared_clone( $_ ), @r; sub A { while ( 1 ) { print Dumper \@s; sleep 3; } } sub B { while ( 1 ) { my $cmd = <STDIN>; chomp $cmd; push @s, { $cmd => time } } } $th1 = threads -> new ( 'A' ) ; $th2 = threads -> new ( 'B' ) ; $th1->join(); $th2->join();
    This kind of assessment ( shareable R/W stuct ) would be more prior for my needs.

    And for my ultimate purpose, I wish to share a struct which carrying different on-the-fly created objects, and they can cross access each other. Of cause I can change the style to share an array to indicate a "queue" and then every objects go inside the queue to look for action. However, this leads me to think, is if there's any transparent way for a complex stuct become shareable

      For the sub{} share stuff, I am still not sure it's necessity.

      Necessity or not -- though it never is -- it is impossible.

      If I give anything try to alter the cloned structure, it give the Thread 2 terminated abnormally: Invalid value for shared scalar error.

      You are trying to add a non-shared hash into a shared array: push @s, { $cmd => time }; ; and that it illegal.

      You need to share() or share_clone that data you are adding:

      push @s, &share( { $cmd => time } }; ## The & before share() is import +ant here; and one of the few times you should ever use it. ## or push @s, &share_clone( { $cmd => time } }; ## or my %hash : shared = ( $cmd => time() ); push @s, \%hash;
      This kind of assessment ( shareable R/W stuct ) would be more prior for my needs.

      Sorry, but I'm having trouble understanding the meaning of that sentence?

      And for my ultimate purpose, I wish to share a struct which carrying different on-the-fly created objects, and they can cross access each other.... However, this leads me to think, is if there's any transparent way for a complex stuct become shareable

      Covering the last sentence first: As demonstrated; it is perfectly possible to shared complex data structures; and to modify them on the fly. But you do have to learn and follow the rules:

      1. Everything except scalars stored as values in shared arrays and hashes, -- ie. any references to nested structures -- must themselves be shared:
        my @array : shared; $array[ 0 ] = 'fred'; ## ok $array[ 1 ] = 12345; ## ok $array[ 2 ] = [ 1,2,3 ] ## NOT OK $array[ 2 ] = &share( [ 1,2,3 ] ); ## OK $array[ 3 ] = { 'a'..'z' }; ## NOT OK my %hash : shared = ( 'a' .. 'z' ); $array[ 3 ] = \%hash; ## OK ...
      2. When making modifications to shared structures; you must lock the parent structure to avoid collisions:
        lock @s; push @s, &share( { $cmd => time() } ); ... lock %{ $s[ 9 ] }; $s[ 9 ]{ $cmd } = $newtime;
      Of cause I can change the style to share an array to indicate a "queue" and then every objects go inside the queue to look for action.

      Shared data structures often seem like a good solution. The problem with them is how does one thread know when one of the other threads has modified something that it needs to look at?

      The typical naive (Java-esque) approach to this problem is some elaborate setup using condition variables and locking and signaling and waiting; ie. synchronisation.

      The problem with that is that it difficult to get right; prone to deadlocks, livelocks and priority inversions; and even when it is programmed correctly, it is usually ponderous because each communication of state -- condition signaling -- requires at least 3 context switches to occur. And with sod's law and competing threads and processes vying for processor time, often many more.

      NB: these problem with synchronisation are not particular to Perl's shared state implementation; but also every other shared state mechanism; Indeed, they are far less prevalent with Perl due to its pretty unique explicitly-shared-only mechanisms; but still prevalent enough to avoid unless there is no alternative. (I've yet to see an application where there wasn't a better (usually queued) solution!)

      Conversely, communications via queues is easy to reason about and get right first time; is not subject to any of the shared-state nasties above; and is efficient and effective because the right thread gets woken automatically when there is something for it to do; and lies dormant consuming no processor time when then isn't.

      Finally, the problem with your application description is that you've decided that you should used shared structures and have defined it in those terms. If you would describe what you actually need to do; rather than how you think you should do it; it would be possible to suggest more effective alternatives; perhaps even point to existing similar examples or generate a small demonstration.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        THANK YOU VERY VERY VERY MUCH Pope!! A very great lesson learned!
        push @s, &share( { $cmd => time } }; ## The & before share() is import +ant here; and one of the few times you should ever use it. ## or push @s, &share_clone( { $cmd => time } }; ## or my %hash : shared = ( $cmd => time() ); push @s, \%hash;
        my @array : shared; $array[ 0 ] = 'fred'; ## ok $array[ 1 ] = 12345; ## ok $array[ 2 ] = [ 1,2,3 ] ## NOT OK $array[ 2 ] = &share( [ 1,2,3 ] ); ## OK $array[ 3 ] = { 'a'..'z' }; ## NOT OK my %hash : shared = ( 'a' .. 'z' ); $array[ 3 ] = \%hash; ## OK
        These examples explained everything that I was missing.

        My real project is hard to explain, as the project scope is still defining, but from what I can understand, the structure is sound like a "online ARPG game". So, there's a World Map, and there's monsters, and players.

        $World = { Players => [ "player1" => { # each key is a player id ID=>"player1", HP=>9999, MP=>2000, coX=>0, coY=>0, Equip => [ qw/xxxSword yyyShield/ ], Magic => [ qw/Earthquake/ ], Invent => { Cure => 1, Poison => 3 } }, "playerx" => { ... } ], Monsters=>{ MonsterA => [ "coX1-coY1", "coX2-coY2", ... ] MonsterB => [ "coX1-coY1", "coX2-coY2", ... ] } };
        So, whenever a player is connected, a new thread created, while everyone will access the same world map. When players walk, they may able to see monster or other players. and their coords changed, which should able to reflect to other players. And by certain time, killed monsters will reborn.

        This is almost the case I am likely to face. Indeed, this is more safe in the "queue" approach, but on the other side, I believe I will create more bugs while coding as the full picture is harder to retrieve.

        However one situation is true here which is fuzzy condition acceptable. Glitch from time race is acceptable in a certain amount of range.

        Indeed, I still can't decide which approach is better for me

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1010892]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (3)
As of 2014-09-21 00:41 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (165 votes), past polls