Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re: Strategy for managing a very large database with Perl

by dws (Chancellor)
on Jun 18, 2010 at 05:55 UTC ( #845310=note: print w/replies, xml ) Need Help??


in reply to Strategy for managing a very large database with Perl

A typical search would be to retrieve all the values of a <variable> for or for a particular year+yday combination.

If those are the typical search patterns, is a relational database the right hammer to reach for? Why not flat files? Appending to a flat file is a quick operation. If you keep one flat file per day, scanning for the subset of records for that day is trivial. If you have to do a full scan through all the data, one file per day is also easy, and allows the scan to be parallelized if you ever need to throw more machines at the problem.

10Tb over 20 years is 500Mb a year, or very roughly 1.5Mb a day. That's not an unreasonable size for a flat file.

  • Comment on Re: Strategy for managing a very large database with Perl

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://845310]
help
Chatterbox?
[erix]: it has TOP, IIRC
[Corion]: erix: Ah. I use where 1 = 0 for queries where I only am interested in the structure ;)
[erix]: 1=0 is as short as TOP :)
[erix]: "code of someone that died" -- kinda nice if your code stops working too
[erix]: hard to implement, hmm
[Corion]: erix: Well, they also seem to have changed the server, or some software, or whatever, and seem to be in the process of changing the DB schema from having the "username" as primary key to something else.
[Corion]: Far too many things being done at once, or maybe only now has it become apparent that nobody knows that piece of software anymore

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (10)
As of 2017-01-23 09:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you watch meteor showers?




    Results (192 votes). Check out past polls.