Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re: Store large hashes more efficiently

by tmharish (Friar)
on Feb 12, 2013 at 14:28 UTC ( #1018378=note: print w/ replies, xml ) Need Help??


in reply to Store large hashes more efficiently

I came across a similar problem and my performance hit was very similar to the one described by Tux.

I got around this by using File::Cache. Considering you seem to be fine with trading run time for memory this might be an option.

A slightly faster but more CPU intensive approach ( that I finally settled on ) was to selectively keep elements in the hash - So essentially the solution was a multi level cache with the first level being the hash and elements in that expire based on either time or frequency of access, and when a cache miss on the hash is generated you look it up in File::Cache - Of course the additional CPU usage comes in when handling the removal of expired hash cache elements.


Comment on Re: Store large hashes more efficiently
Re^2: Store large hashes more efficiently
by puterboy (Scribe) on Feb 12, 2013 at 15:24 UTC
    Interestingly, my previous iteration of the program did something very similar to File::Cache. I created a multi-level directory decimal tree indexed by the most-significant digits of the index. I then stored the values in the leafs of the lowest branches. The value could then be looked up by reading the corresponding file.

    I also implemented some simple caching where a hash stored the most recently referenced values to avoid file accesses where possible.

    However, I found the file accesses to be exceedingly slow (10x penalty) and the cache didn't help me much since there was little correlation between neighboring accesses. In fact, this is what led me to just store everything in a hash since the caching-via-filesystem-tree was too slow. And this is what has led me to try to optimize the storage size of the hash...

    Still fascinating that I did essentially create my own File::Cache like approach before discarding it...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1018378]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (4)
As of 2014-08-30 05:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (291 votes), past polls