Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
As part of my company's PCI requirements, I have been assigned a task to parse a log file that is updated continually through the day. I figure that I can key off the date entry to get just the ones for the previous day, so that I don't duplicate anything, but my big issue is with extracting a specific piece of data. A small sample of the log is below:
2007-11-16 16:04:33 Local1.Alert 128.29.29.40 id=firewall tim +e="2007-11-16 16:04:08" fw=WS2000-Store 29 pri=1 proto=6(tcp) src=128 +.29.29.200 dst=128.29.100.102 mid= 1013 mtp= 2 msg=TCP connection re +quest received is invalid, dropping packet Src 23 Dst 4412 from EXT n +/w agent=Firewall 2007-11-16 16:05:05 Local1.Alert 128.24.24.40 id=firewall tim +e="2007-11-16 16:03:25" fw=WS2000-Store 24 pri=1 proto=6(tcp) src=128 +.24.24.200 dst=128.24.100.101 mid= 1013 mtp= 2 msg=TCP connection re +quest received is invalid, dropping packet Src 23 Dst 4344 from EXT n +/w agent=Firewall 2007-11-16 16:05:34 Local1.Alert 128.29.29.40 id=firewall tim +e="2007-11-16 16:05:09" fw=WS2000-Store 29 pri=1 proto=6(tcp) src=128 +.29.29.200 dst=128.29.100.102 mid= 1013 mtp= 2 msg=TCP connection re +quest received is invalid, dropping packet Src 23 Dst 4412 from EXT n +/w agent=Firewall 2007-11-16 16:05:39 Local1.Alert 128.2.2.40 id=firewall time= +"2007-11-16 16:03:36" fw=WS2000-Store 02 pri=1 proto=6(tcp) src=128.2 +.2.200 dst=128.2.100.106 mid= 1013 mtp= 2 msg=TCP connection request + received is invalid, dropping packet Src 23 Dst 4631 from EXT n/w ag +ent=Firewall 2007-11-16 16:05:40 Local1.Alert 128.2.2.40 id=firewall time= +"2007-11-16 16:03:36" fw=WS2000-Store 02 pri=1 proto=6(tcp) src=128.2 +.2.200 dst=128.2.100.106 mid= 1013 mtp= 2 msg=TCP connection request + received is invalid, dropping packet Src 23 Dst 4631 from EXT n/w ag +ent=Firewall 2007-11-16 16:05:40 Local1.Alert 128.2.2.40 id=firewall time= +"2007-11-16 16:03:37" fw=WS2000-Store 02 pri=1 proto=6(tcp) src=128.2 +.2.200 dst=128.2.100.106 mid= 1013 mtp= 2 msg=TCP connection request + received is invalid, dropping packet Src 23 Dst 4631 from EXT n/w ag +ent=Firewall
I need to extract the data starting with "msg=" and ending just before "Src". The code I am currently using to put the above data into a csv file that I later import to an Excel spreadsheet is below
#!perl use strict; open INPUT,"<","input_file.txt"||die "Can not open input_file: $!\n"; open CSV,">","OUTPUT.csv"||die "Can not open OUTPUT.csv: $!\n"; print CSV "Date,Time,WS 2000,FW Date,FW Time,Store,Src IP,Src Port,Dst + IP,Dst Port,Type,Agent\n"; while(<INPUT>){ my @line = split /\s+/; my $Date = $line[0]; my $Time = $line[1]; my $ws2k = $line[3]; my $FW_Date = $line[5]; my $FW_Time = $line[6]; my $store = $line[8]; my $src_ip = $line[11]; my $dst_ip = $line[12]; my $src_prt = $line[$#line - 6]; my $dst_prt = $line[$#line - 4]; my $type = $line[$#line - 2]; my $agent = $line[$#line]; chomp $agent; $agent=~s/agent=//; $FW_Date=~s/time="//; $FW_Time=~s/"//; $src_ip=~s/src=//; $dst_ip=~s/dst=//; print CSV "$Date,$Time,$ws2k,$FW_Date,$FW_Time,$store,$src_ip,$src_p +rt,$dst_ip,$dst_prt,$type,$agent\n"; } close INPUT; close CSV;
I am guessing that I need to do something along the lines of a regular expression before I actually split the line out into the array. Would something like the following even be close to what I am looking for, or am I heading in the wrong direction with this?
while(<INPUT>){ $_=~m/msg=(.*) Src/; my $msg=$1; my @line = split /\s+/; ## rest of code
As always, thanks for any suggestions.

TStanley
--------
People sleep peaceably in their beds at night only because rough men stand ready to do violence on their behalf. -- George Orwell

In reply to Parsing a log file by TStanley

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others lurking in the Monastery: (4)
    As of 2020-03-30 16:43 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?
      To "Disagree to disagree" means to:









      Results (175 votes). Check out past polls.

      Notices?