Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Epoch based parser

by spikeinc (Acolyte)
on Aug 09, 2013 at 09:29 UTC ( #1048724=perlquestion: print w/ replies, xml ) Need Help??
spikeinc has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

Introducing myself as a very new beginner in perl so sorry for basic questions as always, but I always get the right answer in perlmonks and I love this place...

so heres my problem. I have a file output which I collect using perl. The output contains several lines of logs each with a timestamp. Now I need to process this output and print just one line out of the output. This line tat I want to print is based on the specific time of the day.

The challenge is the time is epoch time. I have to print say for example the line corresponding to 12:15 AM in the logs by parsing each line of epoch time, convert it to date/time format and then if it matches 12:15, print only that line from the log.

My idea was to import the epoch time into array, convert the epoch time to readable time, then do a condition match. But it turned out to be much more complex than that and I couldnt get it work.

Could someone help me. Here is a sample output: the big numbers like 1375976271 are the epoch time start and end. from this output I want to print the line matching a specific time.
{"DisplayRecords":"12","Records":"12","sColumns":"startTime,endTime, remoteNode,srcIP,srcPort,destIP,destPort,egress,ingress","aaData":[["1 +375976271" ,"1375976430","LAN","D0:05:FE","172.20.30.2",1093,"172.20.28.2",1330," +1034,348", "0,0"],["1375976246","1375976404","LAN","D0:05:FE","172.20.30.2",1092, +"172.20.28 .2",1327,"1034,348","0,0"],["1375976191","1375976381","LAN","D0:05:FE" +,"172.20.3 0.2",1091,"172.20.28.2",1324,"1034,348","0,0"],["1375976171","13759763 +14","LAN", "D0:05:FE","172.20.30.2",1090,"172.20.28.2",1321,"1034,348","0,0"],["1 +375976144" ,"1375976302","LAN","D0:05:FE","172.20.30.2",1089,"172.20.28.2",1319," +1034,348", "0,0"],["1375975155","1375976089","LAN","D0:05:FE","172.20.30.2",1085, +"172.20.28 .2",1278,"1034,348","0,0"],["1375975194","1375975352","LAN","D0:05:FE" +,"172.20.3 0.2",1086,"172.20.28.2",1281,"1034,348","0,0"],["1375974693","13759748 +35","LAN", "D0:05:FE","172.20.30.2",1083,"172.20.28.2",1271,"1034,348","0,0"],["1 +375974668" ,"1375974827","LAN","D0:05:FE","172.20.30.2",1081,"172.20.28.2",1268," +1034,348", "0,0"],["1375974691","1375974821","LAN","D0:05:FE","172.20.30.2",1082, +"172.20.28 .2",1270,"0,0","0,0"],["1375881371","1375881561","LAN","D0:05:FE","172 +.20.30.2", 1055,"172.20.28.2",1122,"1034,348","0,0"],["1375881410","1375881540"," +LAN","E0:0 5:AE","172.20.30.2",1056,"172.20.28.2",1125,"0,0","0,0"]]}

Comment on Epoch based parser
Download Code
Re: Epoch based parser
by moritz (Cardinal) on Aug 09, 2013 at 10:32 UTC
Re: Epoch based parser
by RichardK (Priest) on Aug 09, 2013 at 10:45 UTC

    also Time::Piece provides an object with lots of methods make handling dates easier.

Re: Epoch based parser
by kcott (Abbot) on Aug 09, 2013 at 11:00 UTC

    G'day spikeinc,

    Here's a technique for doing what you want.

    I couldn't find any instances of 12:15 in the data you provided; I've used 15:13 as it did appear a few times. You may want localtime instead of the gmtime I've used here (you didn't specify your requirements in this regard). The JSON decoding was straightforward. Check the Time::Piece documentation to see how I've used it and for ways to format the output (if that's what you want).

    #!/usr/bin/env perl -l use strict; use warnings; use JSON; use Time::Piece; my $wanted_hour = 15; my $wanted_minute = 13; my $data = decode_json join '' => map { chomp; $_ } (<DATA>); for (@{$data->{aaData}}) { print "@$_" if is_wanted_time(@$_[0,1]); } sub is_wanted_time { for (@_) { my $t = gmtime $_; return 1 if $t->hour == $wanted_hour && $t->min == $wanted_min +ute; } return 0; } __DATA__ {"DisplayRecords":"12","Records":"12","sColumns":"startTime,endTime, remoteNode,srcIP,srcPort,destIP,destPort,egress,ingress","aaData":[["1 +375976271" ,"1375976430","LAN","D0:05:FE","172.20.30.2",1093,"172.20.28.2",1330," +1034,348", "0,0"],["1375976246","1375976404","LAN","D0:05:FE","172.20.30.2",1092, +"172.20.28 .2",1327,"1034,348","0,0"],["1375976191","1375976381","LAN","D0:05:FE" +,"172.20.3 0.2",1091,"172.20.28.2",1324,"1034,348","0,0"],["1375976171","13759763 +14","LAN", "D0:05:FE","172.20.30.2",1090,"172.20.28.2",1321,"1034,348","0,0"],["1 +375976144" ,"1375976302","LAN","D0:05:FE","172.20.30.2",1089,"172.20.28.2",1319," +1034,348", "0,0"],["1375975155","1375976089","LAN","D0:05:FE","172.20.30.2",1085, +"172.20.28 .2",1278,"1034,348","0,0"],["1375975194","1375975352","LAN","D0:05:FE" +,"172.20.3 0.2",1086,"172.20.28.2",1281,"1034,348","0,0"],["1375974693","13759748 +35","LAN", "D0:05:FE","172.20.30.2",1083,"172.20.28.2",1271,"1034,348","0,0"],["1 +375974668" ,"1375974827","LAN","D0:05:FE","172.20.30.2",1081,"172.20.28.2",1268," +1034,348", "0,0"],["1375974691","1375974821","LAN","D0:05:FE","172.20.30.2",1082, +"172.20.28 .2",1270,"0,0","0,0"],["1375881371","1375881561","LAN","D0:05:FE","172 +.20.30.2", 1055,"172.20.28.2",1122,"1034,348","0,0"],["1375881410","1375881540"," +LAN","E0:0 5:AE","172.20.30.2",1056,"172.20.28.2",1125,"0,0","0,0"]]}

    Output:

    $ pm_epoch_from_json.pl 1375974693 1375974835 LAN D0:05:FE 172.20.30.2 1083 172.20.28.2 1271 1 +034,348 0,0 1375974668 1375974827 LAN D0:05:FE 172.20.30.2 1081 172.20.28.2 1268 1 +034,348 0,0 1375974691 1375974821 LAN D0:05:FE 172.20.30.2 1082 172.20.28.2 1270 0 +,0 0,0

    -- Ken

      Thankyou Ken for saving my day!!

      This is pretty much what I wanted. If I would like to print the output with localtime instead of epoch time (in the output) can I use this function :

       $epochTime=scalar(localtime($Stime));

      Also instead of a specific time mentioned, if I want to filter restults for every one hour (say first 15:13, then 16:13... 17:13 and so on until the end of data) how could I do that...

      I am so thankful to u as I am learning perl a lot, i m a very new beginner so making a lot of mistakes hence hoping that your help will make me proceed. thankyou so much again

        "If I would like to print the output with localtime instead of epoch time (in the output) can I use this function : ..."

        Instead of asking if you can use it, just use it. If it does what you want, you've learnt something; if it doesn't do what you want, you've still learnt something; and, if you make a mistake and get a warning or error message, you've learnt even more. It won't melt your keyboard, explode your mouse or cause your screen to burst into flames. :-)

        "Also instead of a specific time mentioned, if I want to filter restults for every one hour (say first 15:13, then 16:13... 17:13 and so on until the end of data) how could I do that..."

        That should be really obvious from the code I posted. I strongly recommend that, before reading anything else, you study perlintro -- a brief introduction and overview of Perl.

        "... I am learning perl a lot, i m a very new beginner so making a lot of mistakes ..."

        We all make mistakes; even those who've been writing Perl (or, indeed, any language) for decades still mistakes. Mistakes are good: you'll learn a lot from making mistakes. You'll learn nothing from just copying someone else's code (with all the mistakes already fixed).

        As you're clearly very new to Perl, here's some additional documentation links that may further help with the original code I posted (and with any code you write in the future):

        If, after writing some code, you encounter a hurdle you can't resolve yourself, post a question here using the guidelines in "How do I post a question effectively?".

        -- Ken

      Hi Ken,

      Finally I made some changes and then also added code to read the json string from a file and parse it. However, my file is going to contain the same kind of json output appended to it continuously over time.

      When I try to decode_json the entire file, it doesnt allow, as I believe decode_json looks for the braces to see if it finished. How can I parse multiline json from a file?

      Heres the code I have now:

      #!/usr/bin/env perl -l use strict; use warnings; use Time::Local; use JSON; use Time::Piece; use Math::Round; use File::Read; use Text::Table; my $DestIP="192.168.127.111"; my $jsonc = read_file ('C:\Documents and Settings\SonicWALL User\My Do +cuments\Scripts\logs\json.txt'); my $data = decode_json join '' => map { chomp; $_ } ($jsonc); #my $data = decode_json $jsonc; my $tb = Text::Table->new( "Date", "Start Time", "End Time", "Time Taken" , "Egress: Conv +eyed", "Egress : Sent" , "Ingress: Conveyed" , "Ingress: Sent"); for (@{$data->{aaData}}) { my $difftime=(@$_[1] - @$_[0]); my $destip=@$_[6]; my @egress=split(',', @$_[8]); my @ingress=split(',', @$_[9]); my $econveyed=$egress[0]/1000000; my $esent=$egress[1]/1000000; my $iconveyed=$ingress[0]/1000; my $isent=$ingress[1]/1000; my $startTime=scalar(localtime(@$_[0])); my $endTime=scalar(localtime(@$_[1])); #Filter only for required traffic and eliminate any traffic less than + 1MB if ( $destip eq $DestIP && $egress[0]>1000000) { #print "@$_"; my $date = $startTime->mdy; my $stime1 = $startTime->hms; my $etime1 =$endTime->hms; $tb->load( [ $date, $stime1 , $etime1 ,convert_time($difftime) , nearest(.01,$e +conveyed) , nearest(.01,$esent), $iconveyed , $isent ], ); } } print $tb;
      The file input would be something like:
      {"DisplayRecords":"12","Records":"12","sColumns":"startTime,endTime, remoteNode,srcIP,srcPort,destIP,destPort,egress,ingress","aaData":[["1 +375976271" ,"1375976430","LAN","D0:05:FE","172.20.30.2",1093,"172.20.28.2",1330," +1034,348"]]} {"DisplayRecords":"12","Records":"12","sColumns":"startTime,endTime, remoteNode,srcIP,srcPort,destIP,destPort,egress,ingress","aaData":[["1 +375976271" ,"1375976430","LAN","D0:05:FE","172.20.30.2",1093,"172.20.28.2",1330," +1034,348"]]} .... and so on

      I couldnt figure this part out. Thanks for your help so far.

        "When I try to decode_json the entire file, it doesnt allow, as I believe decode_json looks for the braces to see if it finished. How can I parse multiline json from a file?"

        With the sample input you posted, you can just read records as being delimited with "}\n" by setting the input record separator: $/ (see perlvar: Variables related to filehandles). You can then remove the embedded "\n" and "\n+" strings with s/\n[+]?//gm (see perlre if you're unfamiliar with that). Here's a modification of my original code that does this. [Note: you haven't supplied data that matches your original 12:15 or subsequent any:20 — I've made an additional change in order to get some output.]

        #!/usr/bin/env perl -l use strict; use warnings; use JSON; use Time::Piece; my $wanted_minute = 40; { local $/ = "}\n"; while (<DATA>) { s/\n[+]?//gm; my $data = decode_json $_; for (@{$data->{aaData}}) { print "@$_" if is_wanted_time(@$_[0,1]); } } } sub is_wanted_time { for (@_) { my $t = gmtime $_; return 1 if $t->min == $wanted_minute; } return 0; } __DATA__ {"DisplayRecords":"12","Records":"12","sColumns":"startTime,endTime, remoteNode,srcIP,srcPort,destIP,destPort,egress,ingress","aaData":[["1 +375976271" ,"1375976430","LAN","D0:05:FE","172.20.30.2",1093,"172.20.28.2",1330," +1034,348"]]} {"DisplayRecords":"12","Records":"12","sColumns":"startTime,endTime, remoteNode,srcIP,srcPort,destIP,destPort,egress,ingress","aaData":[["1 +375976271" ,"1375976430","LAN","D0:05:FE","172.20.30.2",1093,"172.20.28.2",1330," +1034,348"]]}

        Output:

        $ pm_epoch_from_json_2.pl 1375976271 1375976430 LAN D0:05:FE 172.20.30.2 1093 172.20.28.2 1330 1 +034,348 1375976271 1375976430 LAN D0:05:FE 172.20.30.2 1093 172.20.28.2 1330 1 +034,348
        "The file input would be something like: ..."

        I doubt it!

        That looks like you've just pasted it from the web page including the leading '+'s indicating text wrapping at 70 characters. Assuming that's right, you only need "s/\n//gm" for the regex.

        -- Ken

Re: Epoch based parser
by Perlbotics (Abbot) on Aug 09, 2013 at 11:14 UTC

    Seems you want your HH:MM (localtime or GMT?) to match within the given range?

    use strict; use warnings; use JSON; use Time::Local; my ($hh, $mm) = (15, 20); # 15:20 while ( my $data = <DATA> ) { my $struct = from_json( $data ); if ( my @res = filter_data_hh_mm( $struct, $hh, $mm ) ) { # ... do something print "Matches: $hh:$mm: [ " .gmtime( $_->[0] ) . ".." . gmtime( +$_->[1] ) . " ]\n" for ( @res ); } } sub filter_data_hh_mm { my ($struct, $hh, $mm) = @_; my @res; for my $line ( @{ $struct->{aaData} } ) { # match if begin is before begin-day HH:MM:59 and # if end is after end-day HH:MM:00 if ( $line->[0] <= epoch_at_hhmmss( $line->[0], $hh, $mm , 59) and epoch_at_hhmmss( $line->[1], $hh, $mm , 0) <= $line->[1] ) { push @res, $line; } } return @res; # matching lines or () } sub epoch_at_hhmmss { my ( $epoch, $hh, $mm, $ss ) = @_; #-- use localtime(), timelocal() if $hh:$mm is in localtime # or adjust $hh:$mm to UTC/GMT accordingly my @epoch0 = gmtime( $epoch ); @epoch0[0..2] = ($ss, $mm, $hh); return timegm( @epoch0 ); } __DATA__ {"DisplayRecords":"12","Records":"12","sColumns":"startTime,endTime,re +moteNode,srcIP,srcPort,destIP,destPort,egress,ingress","aaData":[["13 +75976271","1375976430","LAN","D0:05:FE","172.20.30.2",1093,"172.20.28 +.2",1330,"1034,348","0,0"],["1375976246","1375976404","LAN","D0:05:FE +","172.20.30.2",1092,"172.20.28.2",1327,"1034,348","0,0"],["137597619 +1","1375976381","LAN","D0:05:FE","172.20.30.2",1091,"172.20.28.2",132 +4,"1034,348","0,0"],["1375976171","1375976314","LAN","D0:05:FE","172. +20.30.2",1090,"172.20.28.2",1321,"1034,348","0,0"],["1375976144","137 +5976302","LAN","D0:05:FE","172.20.30.2",1089,"172.20.28.2",1319,"1034 +,348","0,0"],["1375975155","1375976089","LAN","D0:05:FE","172.20.30.2 +",1085,"172.20.28.2",1278,"1034,348","0,0"],["1375975194","1375975352 +","LAN","D0:05:FE","172.20.30.2",1086,"172.20.28.2",1281,"1034,348"," +0,0"],["1375974693","1375974835","LAN","D0:05:FE","172.20.30.2",1083, +"172.20.28.2",1271,"1034,348","0,0"],["1375974668","1375974827","LAN" +,"D0:05:FE","172.20.30.2",1081,"172.20.28.2",1268,"1034,348","0,0"],[ +"1375974691","1375974821","LAN","D0:05:FE","172.20.30.2",1082,"172.20 +.28.2",1270,"0,0","0,0"],["1375881371","1375881561","LAN","D0:05:FE", +"172.20.30.2",1055,"172.20.28.2",1122,"1034,348","0,0"],["1375881410" +,"1375881540","LAN","E0:05:AE","172.20.30.2",1056,"172.20.28.2",1125, +"0,0","0,0"]]}

    Result:

    Matches: 15:20: [ Thu Aug 8 15:19:15 2013..Thu Aug 8 15:34:49 2013 +] Matches: 15:20: [ Thu Aug 8 15:19:54 2013..Thu Aug 8 15:22:32 2013 +]

      This is great too! am combining all your inputs and trying to make it work. thanks so much for all your help.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1048724]
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (16)
As of 2014-12-18 17:00 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (58 votes), past polls