Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles

by Marshall (Prior)
on Sep 23, 2013 at 06:44 UTC ( #1055237=note: print w/ replies, xml ) Need Help??


in reply to matching datetimestamps and concatenating data where timestamps match from multiple large datafiles

One way to do this is to adopt a "standard" format that is ASCII sortable.

Example: "2012-10-06 1649".
This is Oct 6, 2012 at 16:49 GMT (year-month-date time).

The leading zero's are important!
"2012-10-6 1649" won't sort in the same ASCII order.

The above format sorts in ASCII order. No problem. That also means that you can compare Date X vs Date Y!.
Add seconds as you wish "1649:05", etc. As long as you have leading zero's, ASCII sort order will work. "1649 05" is fine also as long as the format is consistent with space between the minutes and seconds and leading zero's.

In general, log files should use GMT or what is called UTC time. This is time zone independent.

If you use a DB, same idea: normalize to GMT/UTC time - that is what goes into the DB.

Basically, normalize the date/times to a standard ASCII sortable format using GMT/UTC time. Put that into a DB or not. You can sort/search on these times in ASCII format if you want to. Perl sorts ASCII fields like a rocket. If you have a time zone independent date/time format, the DB will also sort/search it like a rocket.


Comment on Re: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles
Re^2: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles
by Happy-the-monk (Monsignor) on Sep 23, 2013 at 07:12 UTC

    Example: "2012-10-06 1649".

    While you nearly accurately describe the right track, you did not name it.

    ISO 8601 is one of the standards that actually contain a lot of sanity, and there are even a few cpan modules helping you to implement it. 2013-09-23T09:12:55

    Cheers, Sören

    Créateur des bugs mobiles - let loose once, run everywhere.
    (hooked on the Perl Programming language)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1055237]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (10)
As of 2014-12-29 16:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (193 votes), past polls