Re^3: How to share complex data structure in threads ?

[BrowserUk]

For the sub{} share stuff, I am still not sure it's necessity.

Necessity or not -- though it never is -- it is impossible.

If I give anything try to alter the cloned structure, it give the Thread 2 terminated abnormally: Invalid value for shared scalar error.

You are trying to add a non-shared hash into a shared array: push @s, { $cmd => time };; and that it illegal.

You need to share() or share_clone that data you are adding:

push @s, &share( { $cmd => time } }; ## The & before share() is import
+ant here; and one of the few times you should ever use it.

## or

push @s, &share_clone( { $cmd => time } }; 

## or 

my %hash : shared = ( $cmd => time() );
push @s, \%hash;

This kind of assessment ( shareable R/W stuct ) would be more prior for my needs.

Sorry, but I'm having trouble understanding the meaning of that sentence?

And for my ultimate purpose, I wish to share a struct which carrying different on-the-fly created objects, and they can cross access each other.... However, this leads me to think, is if there's any transparent way for a complex stuct become shareable

Covering the last sentence first: As demonstrated; it is perfectly possible to shared complex data structures; and to modify them on the fly. But you do have to learn and follow the rules:

1. Everything except scalars stored as values in shared arrays and hashes, -- ie. any references to nested structures -- must themselves be shared:

   my @array : shared;
   $array[ 0 ] = 'fred'; ## ok
   $array[ 1 ] = 12345; ## ok
   $array[ 2 ] = [ 1,2,3 ] ## NOT OK
   $array[ 2 ] = &share( [ 1,2,3 ] ); ## OK
   $array[ 3 ] = { 'a'..'z' }; ## NOT OK
   my %hash : shared = ( 'a' .. 'z' );
   $array[ 3 ] = \%hash; ## OK
   ...
   

2. When making modifications to shared structures; you must lock the parent structure to avoid collisions:

   lock @s;
   push @s, &share( { $cmd => time() } ); 
   
   ...
   lock %{ $s[ 9 ] };
   $s[ 9 ]{ $cmd } = $newtime;
   

Of cause I can change the style to share an array to indicate a "queue" and then every objects go inside the queue to look for action.

Shared data structures often seem like a good solution. The problem with them is how does one thread know when one of the other threads has modified something that it needs to look at?

The typical naive (Java-esque) approach to this problem is some elaborate setup using condition variables and locking and signaling and waiting; ie. synchronisation.

The problem with that is that it difficult to get right; prone to deadlocks, livelocks and priority inversions; and even when it is programmed correctly, it is usually ponderous because each communication of state -- condition signaling -- requires at least 3 context switches to occur. And with sod's law and competing threads and processes vying for processor time, often many more.

NB: these problem with synchronisation are not particular to Perl's shared state implementation; but also every other shared state mechanism; Indeed, they are far less prevalent with Perl due to its pretty unique explicitly-shared-only mechanisms; but still prevalent enough to avoid unless there is no alternative. (I've yet to see an application where there wasn't a better (usually queued) solution!)

Conversely, communications via queues is easy to reason about and get right first time; is not subject to any of the shared-state nasties above; and is efficient and effective because the right thread gets woken automatically when there is something for it to do; and lies dormant consuming no processor time when then isn't.

Finally, the problem with your application description is that you've decided that you should used shared structures and have defined it in those terms. If you would describe what you actually need to do; rather than how you think you should do it; it would be possible to suggest more effective alternatives; perhaps even point to existing similar examples or generate a small demonstration.

[exilepanda]

THANK YOU VERY VERY VERY MUCH Pope!! A very great lesson learned!

push @s, &share( { $cmd => time } }; ## The & before share() is import
+ant here; and one of the few times you should ever use it.
## or
push @s, &share_clone( { $cmd => time } }; 
## or 
my %hash : shared = ( $cmd => time() );
push @s, \%hash;

my @array : shared;
$array[ 0 ] = 'fred'; ## ok
$array[ 1 ] = 12345; ## ok
$array[ 2 ] = [ 1,2,3 ] ## NOT OK
$array[ 2 ] = &share( [ 1,2,3 ] ); ## OK
$array[ 3 ] = { 'a'..'z' }; ## NOT OK
my %hash : shared = ( 'a' .. 'z' );
$array[ 3 ] = \%hash; ## OK

These examples explained everything that I was missing.

My real project is hard to explain, as the project scope is still defining, but from what I can understand, the structure is sound like a "online ARPG game". So, there's a World Map, and there's monsters, and players.

$World = { 
  Players => [ 
   "player1" => { # each key is a player id
     ID=>"player1", HP=>9999, MP=>2000, coX=>0, coY=>0,
     Equip => [ qw/xxxSword yyyShield/ ],
     Magic => [ qw/Earthquake/ ],
     Invent => { Cure => 1, Poison => 3 } 
   },
   "playerx" => { ... } 
  ],
  Monsters=>{
    MonsterA => [ 
      "coX1-coY1", "coX2-coY2", ...
    ]
    MonsterB => [
      "coX1-coY1", "coX2-coY2", ...
    ]
  }
};

So, whenever a player is connected, a new thread created, while everyone will access the same world map. When players walk, they may able to see monster or other players. and their coords changed, which should able to reflect to other players. And by certain time, killed monsters will reborn.

This is almost the case I am likely to face. Indeed, this is more safe in the "queue" approach, but on the other side, I believe I will create more bugs while coding as the full picture is harder to retrieve.

However one situation is true here which is fuzzy condition acceptable. Glitch from time race is acceptable in a certain amount of range.

Indeed, I still can't decide which approach is better for me

[BrowserUk]

So, whenever a player is connected, a new thread created, while everyone will access the same world map.

Hm. If you have any ambition that your game will support more than a few tens of concurrent players, then I agree with anonymonk, 1 thread per player is the wrong way to go. Imagine your game became moderately popular and you have 1000 players on line at a time. 1000 threads all sitting there consuming memory doing essentially nothing waiting for their user to type something. You'd need a machine with huge memory; but the processing could be done by a 386 class cpu and it would still be mostly idle.Now think about how you would scale that to 10,000 players. You'd need 10 times as much memory but still not much more cpu.

Equally, using an in-memory database would be very dangerous. One errant piece of code or a faulty power supply and bang! your game -- ie. your database -- is gone.

If you are serious about developing this, you need to seriously reconsider your architecture. Your game will need to reside in the database, and the database will need to be resilient and scale. You will need (to allow for) multiple comms servers talking to your player clients; and redundant, resilient, actively distributed database server back ends (or a cluster of machines accessing a distributed filesystem like hadoop).

Even if you want to start small on a single machine, you should separate the comms components and database components of the game and run them as separate processes on that machine and have them communicate through the database/distributed filesystem ONLY. Thus, when and if you need more capacity, you can separate the comms onto a different machine from the DB/game handling. And then if things grow larger, you can increase the number of comms servers and DB servers independently of each other to match your growing requirements.

[exilepanda]

Ah Yes! Thank you very much again, for stopping me here!

This reminds me some yrs ago, I used threads shared hash for a simple chat server for our 20 something ppl dept, the hash queues unread messages... and what periodically happening was a Out of memory error, and the mesgs on the server and the server self gone. I finally have to backup the hash per every minute, and start another script to keep "ping" if the server alive or restart with restore data.

A simple hash could go in this way, I can imagine what is going to happen for such complex structure. Seems I can really ONLY rely on the db/fs now. It's not about accuracy or efficiency, but the basic stability and extendability.