Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

How to share complex data structure in threads ?

by exilepanda (Monk)
on Dec 30, 2012 at 04:36 UTC ( #1010889=perlquestion: print w/ replies, xml ) Need Help??
exilepanda has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks, Would like to know if there's any reference technique or code example tutorial which demonstrate how to share complex data structure for threads.

My expectation is to create a complex stuct ( perhaps include sub{} ) that shareable among threads, however threads::shared seems only serve for simple array, scalar or hash. Is that anything I missed in this module, or is there any module that suits this case ? Thanks in advance.

Comment on How to share complex data structure in threads ?
Select or Download Code
Re: How to share complex data structure in threads ?
by BrowserUk (Pope) on Dec 30, 2012 at 05:40 UTC
    threads::shared seems only serve for simple array, scalar or hash.

    Look again at threads::shared::shared_close(). Eg:

    use threads;; use threads::shared; my @r = ( map+{ map{ $_ => [ 1 .. 10 ] } 'a' .. 'z' }, 1..10 );; pp \@r;; [ { a => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], b => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], c => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], d => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], e => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], f => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], g => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], h => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], i => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], j => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], k => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], l => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "m" => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], n => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], o => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], p => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "q" => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], r => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "s" => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], t => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], u => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], v => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], w => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "x" => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "y" => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], z => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], }, { a => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], ... }, ] my @s : shared = map shared_clone( $_ ), @r; pp \@s; [ # tied threads::shared::tie { # tied threads::shared::tie a => [ # tied threads::shared::tie 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ], b => [ # tied threads::shared::tie 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ], c => [ # tied threads::shared::tie 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ], d => [ # tied threads::shared::tie ... ], ... }, { # tied threads::shared::tie a => [ # tied threads::shared::tie 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ], b => ... } ... ]

    It won't clone subroutines, because that's not possible or logical.

    However, whilst this allows you to share complex data structures, you'll generally find that it doesn't achieve what you think you want to achieve, and you'll probably come back asking why not.

    If you would describe your application, there is usually a better way of achieving it.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Thanks for the lead. For the sub{} share stuff, I am still not sure it's necessity. But consider the following code, If I give anything try to alter the cloned structure, it give the Thread 2 terminated abnormally: Invalid value for shared scalar error.
      $| = 1; use Data::Dumper; use threads;; use threads::shared; my @r = ( map+{ map{ $_ => [ 1 .. 2 ] } 'a' .. 'b' }, 1..2 );; my @s : shared = map shared_clone( $_ ), @r; sub A { while ( 1 ) { print Dumper \@s; sleep 3; } } sub B { while ( 1 ) { my $cmd = <STDIN>; chomp $cmd; push @s, { $cmd => time } } } $th1 = threads -> new ( 'A' ) ; $th2 = threads -> new ( 'B' ) ; $th1->join(); $th2->join();
      This kind of assessment ( shareable R/W stuct ) would be more prior for my needs.

      And for my ultimate purpose, I wish to share a struct which carrying different on-the-fly created objects, and they can cross access each other. Of cause I can change the style to share an array to indicate a "queue" and then every objects go inside the queue to look for action. However, this leads me to think, is if there's any transparent way for a complex stuct become shareable

        For the sub{} share stuff, I am still not sure it's necessity.

        Necessity or not -- though it never is -- it is impossible.

        If I give anything try to alter the cloned structure, it give the Thread 2 terminated abnormally: Invalid value for shared scalar error.

        You are trying to add a non-shared hash into a shared array: push @s, { $cmd => time }; ; and that it illegal.

        You need to share() or share_clone that data you are adding:

        push @s, &share( { $cmd => time } }; ## The & before share() is import +ant here; and one of the few times you should ever use it. ## or push @s, &share_clone( { $cmd => time } }; ## or my %hash : shared = ( $cmd => time() ); push @s, \%hash;
        This kind of assessment ( shareable R/W stuct ) would be more prior for my needs.

        Sorry, but I'm having trouble understanding the meaning of that sentence?

        And for my ultimate purpose, I wish to share a struct which carrying different on-the-fly created objects, and they can cross access each other.... However, this leads me to think, is if there's any transparent way for a complex stuct become shareable

        Covering the last sentence first: As demonstrated; it is perfectly possible to shared complex data structures; and to modify them on the fly. But you do have to learn and follow the rules:

        1. Everything except scalars stored as values in shared arrays and hashes, -- ie. any references to nested structures -- must themselves be shared:
          my @array : shared; $array[ 0 ] = 'fred'; ## ok $array[ 1 ] = 12345; ## ok $array[ 2 ] = [ 1,2,3 ] ## NOT OK $array[ 2 ] = &share( [ 1,2,3 ] ); ## OK $array[ 3 ] = { 'a'..'z' }; ## NOT OK my %hash : shared = ( 'a' .. 'z' ); $array[ 3 ] = \%hash; ## OK ...
        2. When making modifications to shared structures; you must lock the parent structure to avoid collisions:
          lock @s; push @s, &share( { $cmd => time() } ); ... lock %{ $s[ 9 ] }; $s[ 9 ]{ $cmd } = $newtime;
        Of cause I can change the style to share an array to indicate a "queue" and then every objects go inside the queue to look for action.

        Shared data structures often seem like a good solution. The problem with them is how does one thread know when one of the other threads has modified something that it needs to look at?

        The typical naive (Java-esque) approach to this problem is some elaborate setup using condition variables and locking and signaling and waiting; ie. synchronisation.

        The problem with that is that it difficult to get right; prone to deadlocks, livelocks and priority inversions; and even when it is programmed correctly, it is usually ponderous because each communication of state -- condition signaling -- requires at least 3 context switches to occur. And with sod's law and competing threads and processes vying for processor time, often many more.

        NB: these problem with synchronisation are not particular to Perl's shared state implementation; but also every other shared state mechanism; Indeed, they are far less prevalent with Perl due to its pretty unique explicitly-shared-only mechanisms; but still prevalent enough to avoid unless there is no alternative. (I've yet to see an application where there wasn't a better (usually queued) solution!)

        Conversely, communications via queues is easy to reason about and get right first time; is not subject to any of the shared-state nasties above; and is efficient and effective because the right thread gets woken automatically when there is something for it to do; and lies dormant consuming no processor time when then isn't.

        Finally, the problem with your application description is that you've decided that you should used shared structures and have defined it in those terms. If you would describe what you actually need to do; rather than how you think you should do it; it would be possible to suggest more effective alternatives; perhaps even point to existing similar examples or generate a small demonstration.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: How to share complex data structure in threads ?
by Anonymous Monk on Dec 30, 2012 at 14:26 UTC
    Although it may seem "obvious" that each player of an MMPG should be "a thread," that isn't what you should do ... no more than you should have a separate thread for each connected user to a website. Connected users are, say, a TCP/IP port. The game itself is a state-machine. The amount of time required to resolve the effect of anyone's move is a few microseconds: moves can be processed effortlessly one-at-a-time with a queue by one thread. (If you want a Perl analogy, look at POE.) And on this point, save yourself trouble: MMPGs are a dime a dozen and so are engines designed to run them. You don't need to write a Perl program from scratch to do such a thing.
      Thanks for introducing POE. Looking at the interface and which looks really relative to what I may found useful for. Though, I am no doing an online game ( i would consider blender + python if needed ), it just easier for me to explain why I need a complex struct shared by threads. The real project is kind of bio-logic stuff ( and I know nothing about it! ) I possibly to do some work to simulate a "brain" like stuffs but the scope is not yet confirmed.

      All I know is I have to listen to certain range of ports, and signals come in, and the data struct will be altered, after `some external calls` returns values list. The ends up result maybe some like aaatttggg(they said it's protein or sort of) which is totally meaningless for me.. and I don't know how to express any further.. haha!

      So up to this stage, all I am looking for is how to share a complex struct which read/write possible, and start to listen on advise for which approach ( queue / struct ) would be more suitable in such data structure scenario.

      Do you have any "See Also" suggestion like POE ? Thanks again! =)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1010889]
Approved by davido
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (12)
As of 2014-11-26 10:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (167 votes), past polls