Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re: selecting columns from a tab-separated-values file

by johngg (Abbot)
on Jan 21, 2013 at 22:58 UTC ( #1014518=note: print w/replies, xml ) Need Help??

in reply to selecting columns from a tab-separated-values file

You might be better off tolerating the expense of loading your data into a database of some sort once, perhaps SQLite, and thereafter take advantage of the speed of the database which is optimised for data look-ups, using SQL queries to access your columns. DBI and DBD::SQLite would facilitate this.

I hope this is helpful.



  • Comment on Re: selecting columns from a tab-separated-values file

Replies are listed 'Best First'.
Re^2: selecting columns from a tab-separated-values file
by ibm1620 (Scribe) on Jan 22, 2013 at 04:15 UTC
    Interesting idea. Can SQLite handle that capacity? 10^9 rows and about 80GB of data total?
      give a go and you'll find out ;) or you can look here:

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1014518]
[stevieb]: agreed. That's why I said at least a half-dozen. If enough of the different formats are present, the date/time folk may not have to request more. If they do, then at least there was a decent base to start with
[stevieb]: I do date and time transformations in both Perl and Python, but not frequently enough to not have to search for the format params etc ;)

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (4)
As of 2017-04-29 02:42 GMT
Find Nodes?
    Voting Booth?
    I'm a fool:

    Results (531 votes). Check out past polls.