Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Re: Store large hashes more efficiently

by Tux (Abbot)
on Feb 12, 2013 at 07:27 UTC ( #1018312=note: print w/replies, xml ) Need Help??

in reply to Store large hashes more efficiently

With 10 million, the size is getting in your way, and maybe even causes swapping (I have no idea about your process size limits). If that happens, speed will be your first problem.

As you didn't tell other resource limits or process requirements, I'd just wanted to note the I created Tie::Hash::DBD to "fix" a similar problem. In my case, my hash ran into a couple of 100_000 entries, and tieing the hash with DB_File was not a solution, as it could not cope. As I was using a database anyway, I thought I might use it. The hash got a lot slower in the beginning, but the overall process time got halved, and with the option to "keep" the hash in the database, subsequent processes gained a lot.

With how you described your problem, Tie::Hash::DBD will probably not solve your problem at hand, but it might be something to look at when it does.

Enjoy, Have FUN! H.Merijn
  • Comment on Re: Store large hashes more efficiently

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1018312]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (4)
As of 2018-03-18 05:10 GMT
Find Nodes?
    Voting Booth?
    When I think of a mole I think of:

    Results (228 votes). Check out past polls.