Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.

Efficient shared memory - possible? how??

by cnd (Acolyte)
on Feb 20, 2012 at 10:47 UTC ( [id://955034]=perlquestion: print w/replies, xml ) Need Help??

cnd has asked for the wisdom of the Perl Monks concerning the following question:

I want to run a dozen simultaneous ford()'d perl scripts (each with it's own indiviaul processor affinity on a multi-CPU host). I want all of them to have efficient access to a large pool of mostly-static shared memory.

For example - I want to be able for every script to do this:-

print $shared{'hugedata'};

and this:-


but for only 1 copy of all that to live in memory.

I specifically do not want to shuffle copies of stuff around, or to go serializing/unserializing everytying all the time.

Is this possible?

If not - how hard do you think it would be to extract the variable-handling functions out of the perl source, and compile it all into some kind of .xs loadable module?

  • Comment on Efficient shared memory - possible? how??

Replies are listed 'Best First'.
Re: Efficient shared memory - possible? how??
by Corion (Patriarch) on Feb 20, 2012 at 12:24 UTC

    If you are content with not having proper Perl data structures but a raw blob of memory, a good approach to sharing that data among processes (and threads) could be File::Map. This gives you a scalar region in which you can read/write shared data using substr.

    If this falls into "serializing/unserializing everything all the time" territory for you, then there is little you can do about it. Perl data structures are reference counted and you will have to do some very hacky and specialized things to share a Perl data structure between processes. I can imagine that you might be able to create "shared" scalars that point to the shared region via newSV_pv(..., 0). This would tell the Perl memory manager that you want to maintain the storage space allocated to the scalar yourself. But writing to that scalar needs to be done through a special routine that makes sure the sizes still fit - you can't grow or shrink the length of the scalar for example.

      I think that file-mapping is an excellent suggestion because it treats the shared resource “at arm’s length” especially in the sense of avoiding the memory-scribbling that can so easily occur among supposedly well-behaved children who are sharing exactly the same playpen at the same time.   Given that bugs will inevitably occur, even due the to slightest typo, this makes things a bit easier to spot.   (e.g. as in the preceding sentence.)

      You might also seriously consider literally using a shared disk file, knowing that most if not all of the actual access will occur through a shared buffer pool that is helpfully maintained for you by the OS.

Re: Efficient shared memory - possible? how??
by BrowserUk (Patriarch) on Feb 20, 2012 at 12:58 UTC


    • Threads & threads::shared.

      Relatively speed efficient, but somewhat memory hungry.

    • forks & Forks::shared.

      Horribly (speed) inefficient -- reads, writes, updated, deletes, locks, signals et al. are communicated between processes via serialisation and sockets to implement a client/server architecture.

    • Sys::Mmap

      Only does strings. You need to implement structures (hashes or arrays) on top of that module. Perhaps via tie.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

Re: Efficient shared memory - possible? how??
by JavaFan (Canon) on Feb 20, 2012 at 12:01 UTC
    Shared memory is just that, a piece of memory that is shared between processes. There's no build in support to share Perl variables.

    You could make a tied hash, which will store its values in shared memory. Beaware though that when you initiate a piece of shared memory, you will have to supply a size. If you're sticking to storing numerical values, then it's easy. 4 or 8 bytes will do. If you want to share strings, and have them mutable as in Perl, you will have to implement your own double pointer trick (as Perl does).

    Optimizing memory is hard work if your choice of tool is Perl, because Perl usually picks speed over memory when having a choice, and because there's a lot of hidden stuff going on around your back.

    Using threads make it easier to share variables, but Perl threads copy a priory every variable that isn't shared, unlike (Unix) forks which uses copy-on-write. Threads may be the better solution for you, but that's something only you can find out, but doing actual measurement.

Re: Efficient shared memory - possible? how??
by Anonymous Monk on Feb 20, 2012 at 10:57 UTC
      Are you sure those are efficient? Looks like they're "serializing" (or at least duplicating everything) to me, and there doesn't seem to be any discussion about *how* they work?
      $it = $SHARED_CACHE{$ordinal}; ... return $it;

      ford/fork... listent to what I *mean*, not what I *say* :-)

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://955034]
Approved by marto
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others learning in the Monastery: (5)
As of 2024-04-15 02:21 GMT
Find Nodes?
    Voting Booth?

    No recent polls found