Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re: Disk based hash (as opposed to RAM based)

by aufflick (Deacon)
on Oct 08, 2012 at 04:43 UTC ( [id://997739]=note: print w/replies, xml ) Need Help??


in reply to Disk based hash (as opposed to RAM based)

If it was me, for that sort of size of data I wouldn't be using any sort of tied hash. Do you need a hash to pass in to someone else's API? (In which case you couldn't assume they wouldn't force all data into ram at once anyway).

What's the scenario that you need to use it in? What format does your data come in? If you need to do fairly sequential access then there are a lot of storage/access options. If you need a lot of random access I'd probably suggest importing the data into an sqlite database (a one-time process) and then refer to that data from your analysis script. Sqlite will automatically do memory based caching etc. for you so it should be surprisingly fast (and free you from worrying about memory utilisation).

  • Comment on Re: Disk based hash (as opposed to RAM based)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://997739]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others admiring the Monastery: (5)
As of 2024-04-23 16:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found