Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?

Re^2: Store large hashes more efficiently

by puterboy (Scribe)
on Feb 12, 2013 at 15:24 UTC ( #1018389=note: print w/replies, xml ) Need Help??

in reply to Re: Store large hashes more efficiently
in thread Store large hashes more efficiently

Interestingly, my previous iteration of the program did something very similar to File::Cache. I created a multi-level directory decimal tree indexed by the most-significant digits of the index. I then stored the values in the leafs of the lowest branches. The value could then be looked up by reading the corresponding file.

I also implemented some simple caching where a hash stored the most recently referenced values to avoid file accesses where possible.

However, I found the file accesses to be exceedingly slow (10x penalty) and the cache didn't help me much since there was little correlation between neighboring accesses. In fact, this is what led me to just store everything in a hash since the caching-via-filesystem-tree was too slow. And this is what has led me to try to optimize the storage size of the hash...

Still fascinating that I did essentially create my own File::Cache like approach before discarding it...
  • Comment on Re^2: Store large hashes more efficiently

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1018389]
[hippo]: See also Getting Started with Perl for some gentle introductory tutorials.
[thao4]: I don't see where I can post my question. I am french so not good in English
[erix]: go to the bottom of this page: Seekers of Perl Wisdom

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (6)
As of 2018-02-21 09:57 GMT
Find Nodes?
    Voting Booth?
    When it is dark outside I am happiest to see ...

    Results (276 votes). Check out past polls.