Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re:^3 How to share huge data structure between threads?

by LogicalChaos (Beadle)
on Jan 15, 2003 at 05:59 UTC ( [id://227063]=note: print w/replies, xml ) Need Help??


in reply to Re:^2 How to share huge data structure between threads?
in thread How to share huge data structure between threads?

OUCH... Seems like a real dB is the only way to go? What sort of performance are you needing? Do you have large quantities of keys or data associated with the keys?

Have you seen Tie::DBI? I've not used it, but it appears interesting, and might be a quick fit to your application.

Please post your eventual solution back to this thread, as I'm curious what you come up with.

Luck,
Chaos

  • Comment on Re:^3 How to share huge data structure between threads?

Replies are listed 'Best First'.
Re:^4 How to share huge data structure between threads?
by diotalevi (Canon) on Jan 15, 2003 at 14:33 UTC

    Tie::DBI is a memory pig and I wouldn't recomend it for Phoenix's problem. In fact, I'm not happy with this whole tie() idea to begin with for this case. In some previous threads on the same problem Phoenix had some other problems where ze was running out of memory. I'm tending to think ze should either use the straight DB APIs straight - without going through tie() or MLDBM shenanigans.

    In fact... I was thinking of something like Pg (not DBD::Pg) combined with either asynchronous queries or cursors might be a good comprimise.

Re:^4 How to share huge data structure between threads?
by ph0enix (Friar) on Jan 15, 2003 at 09:30 UTC

    This should be some kind od dictionary. As I wrotte in my original post. Data structure of stored value is hash of array of objects.

    Currently only working solution is Tie::RDBM with PostgreSQL database backend. Trying to save data in memory I can't share whole hash which contain values as I described before. Because all server threads must be able to add / delete / modify this data I'm currently looking on solutions based on Tie::RDBM + PostgreSQL and MLDBM + BerkeleyDB. When BerkeleyDB with MLDBM is used I get some strange errors probably originaly from Storable module - 'inbalid byte order', 'wrong signature' or 'bad data'...

    Other problem I have with tied hashes and Storable (not solved yet) was described in node Storable: where is my memory?.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://227063]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others admiring the Monastery: (4)
As of 2024-07-12 19:36 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?
    erzuuli‥ 🛈The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.