comment on

As part of my company's PCI requirements, I have been assigned a task to parse a log file that is updated continually through the day. I figure that I can key off the date entry to get just the ones for the previous day, so that I don't duplicate anything, but my big issue is with extracting a specific piece of data. A small sample of the log is below:

2007-11-16 16:04:33    Local1.Alert    128.29.29.40    id=firewall tim
+e="2007-11-16 16:04:08" fw=WS2000-Store 29 pri=1 proto=6(tcp) src=128
+.29.29.200 dst=128.29.100.102 mid= 1013 mtp=  2 msg=TCP connection re
+quest received is invalid, dropping packet Src 23 Dst 4412 from EXT n
+/w agent=Firewall
2007-11-16 16:05:05    Local1.Alert    128.24.24.40    id=firewall tim
+e="2007-11-16 16:03:25" fw=WS2000-Store 24 pri=1 proto=6(tcp) src=128
+.24.24.200 dst=128.24.100.101 mid= 1013 mtp=  2 msg=TCP connection re
+quest received is invalid, dropping packet Src 23 Dst 4344 from EXT n
+/w agent=Firewall
2007-11-16 16:05:34    Local1.Alert    128.29.29.40    id=firewall tim
+e="2007-11-16 16:05:09" fw=WS2000-Store 29 pri=1 proto=6(tcp) src=128
+.29.29.200 dst=128.29.100.102 mid= 1013 mtp=  2 msg=TCP connection re
+quest received is invalid, dropping packet Src 23 Dst 4412 from EXT n
+/w agent=Firewall
2007-11-16 16:05:39    Local1.Alert    128.2.2.40    id=firewall time=
+"2007-11-16 16:03:36" fw=WS2000-Store 02 pri=1 proto=6(tcp) src=128.2
+.2.200 dst=128.2.100.106 mid= 1013 mtp=  2 msg=TCP connection request
+ received is invalid, dropping packet Src 23 Dst 4631 from EXT n/w ag
+ent=Firewall
2007-11-16 16:05:40    Local1.Alert    128.2.2.40    id=firewall time=
+"2007-11-16 16:03:36" fw=WS2000-Store 02 pri=1 proto=6(tcp) src=128.2
+.2.200 dst=128.2.100.106 mid= 1013 mtp=  2 msg=TCP connection request
+ received is invalid, dropping packet Src 23 Dst 4631 from EXT n/w ag
+ent=Firewall
2007-11-16 16:05:40    Local1.Alert    128.2.2.40    id=firewall time=
+"2007-11-16 16:03:37" fw=WS2000-Store 02 pri=1 proto=6(tcp) src=128.2
+.2.200 dst=128.2.100.106 mid= 1013 mtp=  2 msg=TCP connection request
+ received is invalid, dropping packet Src 23 Dst 4631 from EXT n/w ag
+ent=Firewall
[download]

I need to extract the data starting with "msg=" and ending just before "Src". The code I am currently using to put the above data into a csv file that I later import to an Excel spreadsheet is below

#!perl
use strict;

open INPUT,"<","input_file.txt"||die "Can not open input_file: $!\n";
open CSV,">","OUTPUT.csv"||die "Can not open OUTPUT.csv: $!\n";

print CSV "Date,Time,WS 2000,FW Date,FW Time,Store,Src IP,Src Port,Dst
+ IP,Dst Port,Type,Agent\n";

while(<INPUT>){
  my @line = split /\s+/;
      
  my $Date    = $line[0];
  my $Time    = $line[1];
  my $ws2k    = $line[3];
  my $FW_Date = $line[5];
  my $FW_Time = $line[6];
  my $store   = $line[8];
  my $src_ip  = $line[11];
  my $dst_ip  = $line[12];
  my $src_prt = $line[$#line - 6];  
  my $dst_prt = $line[$#line - 4];
  my $type    = $line[$#line - 2];
  my $agent   = $line[$#line];

  chomp $agent;
  $agent=~s/agent=//;
  $FW_Date=~s/time="//;
  $FW_Time=~s/"//;
  $src_ip=~s/src=//;
  $dst_ip=~s/dst=//;
    
  print CSV "$Date,$Time,$ws2k,$FW_Date,$FW_Time,$store,$src_ip,$src_p
+rt,$dst_ip,$dst_prt,$type,$agent\n";
}

close INPUT;
close CSV;
[download]

I am guessing that I need to do something along the lines of a regular expression before I actually split the line out into the array. Would something like the following even be close to what I am looking for, or am I heading in the wrong direction with this?

while(<INPUT>){
  $_=~m/msg=(.*) Src/;
  my $msg=$1;

  my @line = split /\s+/;  
  ## rest of code
[download]

As always, thanks for any suggestions.

TStanley
--------
People sleep peaceably in their beds at night only because rough men stand ready to do violence on their behalf. -- George Orwell

In reply to Parsing a log file by TStanley

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		`&`
	<		`<`
	>		`>`
	[		`[`
	]		`]`

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.


Syntactic Confectionery Delight
	PerlMonks