Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles

by wjw (Priest)
on Sep 22, 2013 at 20:24 UTC ( #1055199=note: print w/replies, xml ) Need Help??


in reply to matching datetimestamps and concatenating data where timestamps match from multiple large datafiles

I have to agree with the database approach brought up earlier. I occasionally run into these issues where correlation between large files is required and have yet to be able to beat the combo of perl and DBI/DBD. SQLite has been a life-saver

  • for ease of use
  • quick implementation
  • a short learning curve
  • as well as performance savings

Best of luck in whatever approach you use...

  • ...the majority is always wrong, and always the last to know about it...
  • The Spice must flow...
  • ..by my will, and by will alone.. I set my mind in motion
  • Comment on Re: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1055199]
help
Chatterbox?
[erix]: aka: create extension if not exists file_fdw;
[erix]: the annoying thing is that during setup of the 'foreign table' you have to specify all the columns. But that is reasonably automatable (just reading header line, and contruct the needed CREATE FOREIGN TABLE' statement)
[erix]: it may be easier to just slurp your fils into regular tables. Has other infrastructure advantages also (indexes, constraints etc)

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (5)
As of 2018-02-23 15:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    When it is dark outside I am happiest to see ...














    Results (302 votes). Check out past polls.

    Notices?