Converting datetime string to valid localtime value (seconds)

[vladb]

While working on a small log monitoring utility, I encountered a few issues trying to convert datetime string appearing in a log entry to a valid localtime value such that localtime($logtime_in_seconds) would represent the exact time the log entry was made.

I've played with a number of Date:: modules to no avail. One particular module that I thought would help is Date::Manip. Here's the code snippet that demonstrates the problem (or shell I call it a 'bug'?):

use strict;
use Date::Manip;

my $line = qq{1.2.3.4 - - [15/May/2003:01:05:02 -0600] "GET /foobar"};

my( @mon ) = ( "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug",
+ "Sep" , "Oct", "Nov", "Dec" );
my $i = 0;
my %mon = map { $_ => ++$i } @mon;

if ($aline =~ m/
        ([\d\.]+)   # IP        
        .*?\[([^\s]+?)\s  # date        
         ([-+]\d+)  # zone
         \]\s
         (.*)       # text
         /x) 
 {

    my ($ip, $date, $zone, $text) = ($1, $2, $3, $4);    
    print "($ip, $date, $zone, $text)\n";
    my ($d, $mon, $timestr) = split("/", $date);
    $d =~ s/^0//g; 
    my ($y, $h, $mn, $s) = split(":", $timestr);

    # calculate zone offset (in seconds)
    my $zf = ($zone =~ m/^-/) ? -1 : 1; # zone factor
    my @z = split("",$zone); shift @z; 
    my $zsec = $z[0]*216000+$z[1]*3600+$z[2]*60+$z[3];

    print "Log date values: $mon{$mon}, $d, $y, $h, $mn, $s\n";
    print "*** ERROR ***>> Date_SecsSince1970($mon{$mon}, $d, $y, $h, 
+$mn, $s) = " . 
        localtime(Date_SecsSince1970($mon{$mon}, $d, $y, $h, $mn, $s))
+ . " <<***\n";

    my $time = Date_SecsSince1970($mon{$mon}, $d, $y, $h, $mn, $s) + $
+zf*$zsec;
    print "parsed time: ".localtime($time)."; original: $date $zone\n"
+;
}

This produces the following output:

(1.2.3.4, 15/May/2003:01:05:02, -0600, "GET /foobar")
Log date values: 5, 15, 2003, 01, 05, 02
*** ERROR ***>> Date_SecsSince1970(5, 15, 2003, 01, 05, 02) = Wed May 
+14 18:05:02 2003 <<***
parsed time: Wed May 14 12:05:02 2003; original: 15/May/2003:01:05:02 
+-0600

Note the *** ERROR *** line. While the log date value is 15 (of May), the Date_SecsSince1970() method converts it to 14th (or, to be precise, it returns an invalid second count value by, apparently, missing a day). You may disregard the hour value as this may be related to time zone offset calculations (the $zsec variable).

Having played with this module for awhile, I now feel it may not be the right tool in this case. Has any of you come across similar issues? What would be the best module / approach to use?

_____________________
"We've all heard that a million monkeys banging on a million typewriters will eventually reproduce
the entire works of Shakespeare. Now, thanks to the Internet, we know this is not true."

Robert Wilensky, University of California

[grep]

Hope fully this helps.

#!/usr/local/bin/perl
use strict;
use Date::Manip;

my $line = qq{1.2.3.4 - - [15/May/2003:01:05:02 -0600] "GET /foobar"};

my $date = '';

if ($line =~ /\[([^\]]+)\]/) {
    $date = $1;
}

my $secs = UnixDate(ParseDate($date),"%s");
print "$secs\n";
print localtime($secs)."\n";

You can also throw the DateCalc sub in there for your time zone.

UPDATE: Better solution


$Date::Manip::TZ = '-0800';
my $line = qq{1.2.3.4 - - [15/May/2003:01:05:02 -0600] "GET /foobar"};

my $date = '';

if ($line =~ /\[([^\]]+)\]/) {
    $date = $1;
}
my $tz = (split(/\s+/,$date))[1];

print "$date $tz\n";
my $secs = UnixDate(Date_ConvTZ(ParseDate($date),$tz,'GMT' ),"%s");
print "$secs\n";
print localtime($secs)."\n";



grep

Mynd you, mønk bites Kan be pretti nasti...

[vladb]

grep++, thanks this has worked wonderfully! ;-)

My code is a prime example of how one should never misuse (or shell I say 'abuse'?) a module for his own good. {grin}

update:
Thanks for the timezone conversion code, grep! ;) Although now I run this monitor script on the same server that the primary application (generating the logs) is running on, there's a possiblity of monitoring logs generated on remote servers residing in different time zones. This is when the timezone conversion will become handy.

_____________________
"We've all heard that a million monkeys banging on a million typewriters will eventually reproduce
the entire works of Shakespeare. Now, thanks to the Internet, we know this is not true."

Robert Wilensky, University of California

[halley]

Off by a day? Without digging deeper, I recall that the arrays returned by localtime() have a month from 0 to 11, inexplicably, but a day from 1 to ~31. Whether that's what you're hitting, I thought I'd take this opportunity to gripe about such fencepost errors.

--
[ e d @ h a l l e y . c c ]

[vladb]

How do array indexes come into play when I'm only looking at this line of code:

localtime(Date_SecsSince1970($mon{$mon}, $d, $y, $h, $mn, $s))

If the values of $mon{$mon}, $d, $y, $h are valid (5, 15, 2003, 01 respectively), why the resulting datetime string returned by localtime() is in fact "Wed May 14 18:05:02 2003"?

Please note that I do not force localtime() to return datetime values in array format. What this line of code (and accompanying output) seem to indicate is that the Date_SecsSince1970() method miscalculates seconds or, to state it otherwise, calculates seconds in a way that is incompatible with how the time() method does it.

Afterall, print "time string: " . localtime(time()) would produce valid result.

_____________________
"We've all heard that a million monkeys banging on a million typewriters will eventually reproduce
the entire works of Shakespeare. Now, thanks to the Internet, we know this is not true."

Robert Wilensky, University of California

[Anonymous Monk]

For those who got here (like me) who want to reverse a *string* created by scalar(localtime(time())), the following example shows how to do it:

use Time::Local;
@t = split(/[ :]/,scalar(localtime(time)));
%mon2num = qw(jan 0 feb 1 mar 2 apr 3 may 4 jun 5 jul 6 aug 7 sep 8 oc
+t 9 nov 10 dec 11);
$mon = $mon2num{lc(substr($t[1],0,3))};
print(scalar(localtime(time())),"\n",
      scalar(localtime(timelocal($t[5],$t[4],$t[3],$t[2],$mon,$t[6])))
+,"\n");'
Mon Oct 29 15:59:06 2018
Mon Oct 29 15:59:06 2018

[hippo]

Alternatively, don't parse the monthnames yourself and use the core module Time::Piece instead:

#!/usr/bin/env perl 
use strict;
use warnings;
use Time::Piece;

my $source = 'Mon Oct 29 15:59:06 2018';
my $time   = Time::Piece->strptime ($source, '%a %b %d %T %Y');

# Now use it however you wish
print $time->datetime . "\n";
print "Today is " . $time->fullday . "\n";