Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re: Store large hashes more efficiently

by Tux (Monsignor)
on Feb 12, 2013 at 07:27 UTC ( #1018312=note: print w/ replies, xml ) Need Help??


in reply to Store large hashes more efficiently

With 10 million, the size is getting in your way, and maybe even causes swapping (I have no idea about your process size limits). If that happens, speed will be your first problem.

As you didn't tell other resource limits or process requirements, I'd just wanted to note the I created Tie::Hash::DBD to "fix" a similar problem. In my case, my hash ran into a couple of 100_000 entries, and tieing the hash with DB_File was not a solution, as it could not cope. As I was using a database anyway, I thought I might use it. The hash got a lot slower in the beginning, but the overall process time got halved, and with the option to "keep" the hash in the database, subsequent processes gained a lot.

With how you described your problem, Tie::Hash::DBD will probably not solve your problem at hand, but it might be something to look at when it does.


Enjoy, Have FUN! H.Merijn


Comment on Re: Store large hashes more efficiently

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1018312]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (3)
As of 2014-11-28 03:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (192 votes), past polls