Efficient shared memory - possible? how??

[cnd]

I want to run a dozen simultaneous ford()'d perl scripts (each with it's own indiviaul processor affinity on a multi-CPU host). I want all of them to have efficient access to a large pool of mostly-static shared memory.

For example - I want to be able for every script to do this:-

print $shared{'hugedata'};

and this:-

$shared{'totalrequests'}++;

but for only 1 copy of all that to live in memory.

I specifically do not want to shuffle copies of stuff around, or to go serializing/unserializing everytying all the time.

Is this possible?

If not - how hard do you think it would be to extract the variable-handling functions out of the perl source, and compile it all into some kind of .xs loadable module?

[Corion]

If you are content with not having proper Perl data structures but a raw blob of memory, a good approach to sharing that data among processes (and threads) could be File::Map. This gives you a scalar region in which you can read/write shared data using substr.

If this falls into "serializing/unserializing everything all the time" territory for you, then there is little you can do about it. Perl data structures are reference counted and you will have to do some very hacky and specialized things to share a Perl data structure between processes. I can imagine that you might be able to create "shared" scalars that point to the shared region via newSV_pv(..., 0). This would tell the Perl memory manager that you want to maintain the storage space allocated to the scalar yourself. But writing to that scalar needs to be done through a special routine that makes sure the sizes still fit - you can't grow or shrink the length of the scalar for example.

[sundialsvc4]

I think that file-mapping is an excellent suggestion because it treats the shared resource “at arm’s length” especially in the sense of avoiding the memory-scribbling that can so easily occur among supposedly well-behaved children who are sharing exactly the same playpen at the same time.   Given that bugs will inevitably occur, even due the to slightest typo, this makes things a bit easier to spot.   (e.g. as in the preceding sentence.)

You might also seriously consider literally using a shared disk file, knowing that most if not all of the actual access will occur through a shared buffer pool that is helpfully maintained for you by the OS.

[BrowserUk]

Options:

• Threads & threads::shared.

  Relatively speed efficient, but somewhat memory hungry.

• forks & Forks::shared.

  Horribly (speed) inefficient -- reads, writes, updated, deletes, locks, signals et al. are communicated between processes via serialisation and sockets to implement a client/server architecture.

• Sys::Mmap

  Only does strings. You need to implement structures (hashes or arrays) on top of that module. Perhaps via tie.

[JavaFan]

Shared memory is just that, a piece of memory that is shared between processes. There's no build in support to share Perl variables.

You could make a tied hash, which will store its values in shared memory. Beaware though that when you initiate a piece of shared memory, you will have to supply a size. If you're sticking to storing numerical values, then it's easy. 4 or 8 bytes will do. If you want to share strings, and have them mutable as in Perl, you will have to implement your own double pointer trick (as Perl does).

Optimizing memory is hard work if your choice of tool is Perl, because Perl usually picks speed over memory when having a choice, and because there's a lot of hidden stuff going on around your back.

Using threads make it easier to share variables, but Perl threads copy a priory every variable that isn't shared, unlike (Unix) forks which uses copy-on-write. Threads may be the better solution for you, but that's something only you can find out, but doing actual measurement.

[Anonymous Monk]

you can't ford()'d anything

use threads::shared, use forks::shared

[cnd]

Are you sure those are efficient? Looks like they're "serializing" (or at least duplicating everything) to me, and there doesn't seem to be any discussion about *how* they work?

        $it = $SHARED_CACHE{$ordinal};
...
    return $it;

ford/fork... listent to what I *mean*, not what I *say* :-)