http://www.perlmonks.org?node_id=225889


in reply to How to share huge data structure between threads?

Well, you don't say how big huge is, so...

Have you looked into IPC::Shareable? You can tie your hash to the shared memory region, and then just make sure you lock it appropriatly before reading/writing. I use this in multi process (not threads) programs and it works quite well for me.

If the standard shared memory segment size is too small (32Mb?), you can increase it runtime by adjusting /proc/sys/kernel/shmmax or re-building the kernel

Cheers,
Rob
  • Comment on Re: How to share huge data structure between threads?

Replies are listed 'Best First'.
Re:^2 How to share huge data structure between threads?
by ph0enix (Friar) on Jan 13, 2003 at 08:49 UTC

     Well, you don't say how big huge is, so...

    Server I'm testing on have 3GB RAM which will be probably insufficient for final application. Current size of the test data I want to share is about 600MB (after loading to perl). I don't think that IPC::Shareable can fit this requirements.

[reply]

      One thought. If your reluctant to move to using a file-based data sharing option (like Tie::DBI or other) because of speed, you might consider creating a Ramdisk and placing the tied file on there.

      Examine what is said, not who speaks.

      The 7th Rule of perl club is -- pearl clubs are easily damaged. Use a diamond club instead.

[reply]
      OUCH... Seems like a real dB is the only way to go? What sort of performance are you needing? Do you have large quantities of keys or data associated with the keys?

      Have you seen Tie::DBI? I've not used it, but it appears interesting, and might be a quick fit to your application.

      Please post your eventual solution back to this thread, as I'm curious what you come up with.

      Luck,
      Chaos

[reply]

        Tie::DBI is a memory pig and I wouldn't recomend it for Phoenix's problem. In fact, I'm not happy with this whole tie() idea to begin with for this case. In some previous threads on the same problem Phoenix had some other problems where ze was running out of memory. I'm tending to think ze should either use the straight DB APIs straight - without going through tie() or MLDBM shenanigans.

        In fact... I was thinking of something like Pg (not DBD::Pg) combined with either asynchronous queries or cursors might be a good comprimise.

[reply]

        This should be some kind od dictionary. As I wrotte in my original post. Data structure of stored value is hash of array of objects.

        Currently only working solution is Tie::RDBM with PostgreSQL database backend. Trying to save data in memory I can't share whole hash which contain values as I described before. Because all server threads must be able to add / delete / modify this data I'm currently looking on solutions based on Tie::RDBM + PostgreSQL and MLDBM + BerkeleyDB. When BerkeleyDB with MLDBM is used I get some strange errors probably originaly from Storable module - 'inbalid byte order', 'wrong signature' or 'bad data'...

        Other problem I have with tied hashes and Storable (not solved yet) was described in node Storable: where is my memory?.

[reply]
Re: Re: How to share huge data structure between threads?
by Jeppe (Monk) on Jan 13, 2003 at 18:31 UTC
    As a sidenote, I just got burned at IPC::Shareable. Not badly. I just failed to RTFM, and discovered the rather hard way the 64K default size of shared memory "segments" or "partitions" or whatever. Read the manual, and look at the size option when tie-ing.
[reply]
      Opps. Sorry, I forgot about mentioning that one. I remember hit the same problem when I first used IPC::Shareable... And I think it took me a day or two to finally read the manual.

      But now you've got lots of memory to use, up to the kernel set limit ;-)

      Cheers,
      Logical
[reply]

In Section Seekers of Perl Wisdom