Highly efficient variable-sharing among processesby cnd (Acolyte)
|on Aug 27, 2016 at 21:44 UTC||Need Help??|
cnd has asked for the wisdom of the Perl Monks concerning the following question:
My unsolved quest for efficient variable(memory) sharing on linux continues.
I have a monster (lots of gigs, upto 200gb in future) in-memory hash and a nice perl sub which does lookups and returns results for me.
I want *other* processes to be able to do lookups as well, obviously without every process loading it's own duplicate copy of the hash.
The rate of lookups is extreme, *many* millions per minute at least.
Can anyone think of an efficient way to code this? The "lookup" child processes will be mod_perl (web cgi); I'm hoping the giant-hash process can simply be a local daemon.
I'm wanting to have zero unnecessary overhead: so: no interprocess communication if possible (no sockets, no messages), no memory-shuffling (I don't want to move gigs of hash out of a shared memory pool into the web server pool prior to starting the lookup), no serializing...
Has anyone ever built a modern perl from source? What about the idea of modding the source to introduce some kind of shared region that way? e.g. every variable with the name "shared_*" uses a different subset of RAM or something like that?