As part of my company's PCI requirements, I have been assigned a task to parse a log file that is updated continually through the day. I figure that I can key off the date entry to get just the ones for the previous day, so that I don't duplicate anything, but my big issue is with extracting a specific piece of data. A small sample of the log is below:

2007-11-16 16:04:33    Local1.Alert    128.29.29.40    id=firewall tim
+e="2007-11-16 16:04:08" fw=WS2000-Store 29 pri=1 proto=6(tcp) src=128
+.29.29.200 dst=128.29.100.102 mid= 1013 mtp=  2 msg=TCP connection re
+quest received is invalid, dropping packet Src 23 Dst 4412 from EXT n
+/w agent=Firewall
2007-11-16 16:05:05    Local1.Alert    128.24.24.40    id=firewall tim
+e="2007-11-16 16:03:25" fw=WS2000-Store 24 pri=1 proto=6(tcp) src=128
+.24.24.200 dst=128.24.100.101 mid= 1013 mtp=  2 msg=TCP connection re
+quest received is invalid, dropping packet Src 23 Dst 4344 from EXT n
+/w agent=Firewall
2007-11-16 16:05:34    Local1.Alert    128.29.29.40    id=firewall tim
+e="2007-11-16 16:05:09" fw=WS2000-Store 29 pri=1 proto=6(tcp) src=128
+.29.29.200 dst=128.29.100.102 mid= 1013 mtp=  2 msg=TCP connection re
+quest received is invalid, dropping packet Src 23 Dst 4412 from EXT n
+/w agent=Firewall
2007-11-16 16:05:39    Local1.Alert    128.2.2.40    id=firewall time=
+"2007-11-16 16:03:36" fw=WS2000-Store 02 pri=1 proto=6(tcp) src=128.2
+.2.200 dst=128.2.100.106 mid= 1013 mtp=  2 msg=TCP connection request
+ received is invalid, dropping packet Src 23 Dst 4631 from EXT n/w ag
+ent=Firewall
2007-11-16 16:05:40    Local1.Alert    128.2.2.40    id=firewall time=
+"2007-11-16 16:03:36" fw=WS2000-Store 02 pri=1 proto=6(tcp) src=128.2
+.2.200 dst=128.2.100.106 mid= 1013 mtp=  2 msg=TCP connection request
+ received is invalid, dropping packet Src 23 Dst 4631 from EXT n/w ag
+ent=Firewall
2007-11-16 16:05:40    Local1.Alert    128.2.2.40    id=firewall time=
+"2007-11-16 16:03:37" fw=WS2000-Store 02 pri=1 proto=6(tcp) src=128.2
+.2.200 dst=128.2.100.106 mid= 1013 mtp=  2 msg=TCP connection request
+ received is invalid, dropping packet Src 23 Dst 4631 from EXT n/w ag
+ent=Firewall
[download]

I need to extract the data starting with "msg=" and ending just before "Src". The code I am currently using to put the above data into a csv file that I later import to an Excel spreadsheet is below

#!perl
use strict;

open INPUT,"<","input_file.txt"||die "Can not open input_file: $!\n";
open CSV,">","OUTPUT.csv"||die "Can not open OUTPUT.csv: $!\n";

print CSV "Date,Time,WS 2000,FW Date,FW Time,Store,Src IP,Src Port,Dst
+ IP,Dst Port,Type,Agent\n";

while(<INPUT>){
  my @line = split /\s+/;
      
  my $Date    = $line[0];
  my $Time    = $line[1];
  my $ws2k    = $line[3];
  my $FW_Date = $line[5];
  my $FW_Time = $line[6];
  my $store   = $line[8];
  my $src_ip  = $line[11];
  my $dst_ip  = $line[12];
  my $src_prt = $line[$#line - 6];  
  my $dst_prt = $line[$#line - 4];
  my $type    = $line[$#line - 2];
  my $agent   = $line[$#line];

  chomp $agent;
  $agent=~s/agent=//;
  $FW_Date=~s/time="//;
  $FW_Time=~s/"//;
  $src_ip=~s/src=//;
  $dst_ip=~s/dst=//;
    
  print CSV "$Date,$Time,$ws2k,$FW_Date,$FW_Time,$store,$src_ip,$src_p
+rt,$dst_ip,$dst_prt,$type,$agent\n";
}

close INPUT;
close CSV;
[download]

I am guessing that I need to do something along the lines of a regular expression before I actually split the line out into the array. Would something like the following even be close to what I am looking for, or am I heading in the wrong direction with this?

while(<INPUT>){
  $_=~m/msg=(.*) Src/;
  my $msg=$1;

  my @line = split /\s+/;  
  ## rest of code
[download]

As always, thanks for any suggestions.

---

---

• Are you posting in the right place? Check out Where do I post X? to know for sure.
• Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:

  <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>

• Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
• Want more info? How to link or How to display code and escape characters are good places to start.