Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Re: Store large hashes more efficiently

by Tux (Abbot)
on Feb 12, 2013 at 07:27 UTC ( #1018312=note: print w/replies, xml ) Need Help??


in reply to Store large hashes more efficiently

With 10 million, the size is getting in your way, and maybe even causes swapping (I have no idea about your process size limits). If that happens, speed will be your first problem.

As you didn't tell other resource limits or process requirements, I'd just wanted to note the I created Tie::Hash::DBD to "fix" a similar problem. In my case, my hash ran into a couple of 100_000 entries, and tieing the hash with DB_File was not a solution, as it could not cope. As I was using a database anyway, I thought I might use it. The hash got a lot slower in the beginning, but the overall process time got halved, and with the option to "keep" the hash in the database, subsequent processes gained a lot.

With how you described your problem, Tie::Hash::DBD will probably not solve your problem at hand, but it might be something to look at when it does.


Enjoy, Have FUN! H.Merijn
  • Comment on Re: Store large hashes more efficiently

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1018312]
help
Chatterbox?
[choroba]: No, if there are no threads, then the user wants to use MCE. If there are threads, the user can choose.
[1nickt]: choroba Understood. I'm wondering about the logic in a cpanfile. If the perl doesn't support threads, it's easy: require MCE. If the perl does support threads, as you say the user has a choice, so require both? Or, assume that irrespective of the choide
[1nickt]: ... choice, the threaded-perl user *probably* wants to use threads, therefore only require threads and leave MCE out, allowing the user to manually install if desired?
[choroba]: just require any of MCE and threads. Not sure if that's possible in the cpanfile, but should be possible early in the Makefile
[1nickt]: The app expects threads as default anyway, right?
[choroba]: that's true
[1nickt]: expects threads *to be* the default

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (12)
As of 2017-10-18 13:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My fridge is mostly full of:

















    Results (244 votes). Check out past polls.

    Notices?