Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re: index for large text file

by GrandFather (Sage)
on Mar 28, 2011 at 09:32 UTC ( #895872=note: print w/ replies, xml ) Need Help??


in reply to index for large text file

Would this data be better stored in a database? To answer that you need to think about how the files are generated and used. If you look them up much more often than they are generated a database may be very worth while. If relatively small numbers of records change from time to time that may be another good reason to use a database. If you have control over generation of the current file that will make using a database easier.

True laziness is hard work


Comment on Re: index for large text file
Re^2: index for large text file
by Anonymous Monk on May 16, 2013 at 23:08 UTC
    I do not think a database would be appropriate considering this file type (FASTQ) can easily have 200,000,000 records (200,000,000 X 4 lines). We (biologists) can often have 100's if not 1,000s of these files.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://895872]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (10)
As of 2015-07-07 07:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (87 votes), past polls