Re: Parsing Apache logs with Regex

[kyle]

Works fine for me.

use strict;
use warnings;
#use diagnostics;

my $log_line = '67.60.185.31 - - [14/Jan/2008:02:25:54 -0800] "GET /di
+splay.cgi?2643943|3334115 HTTP/1.1" 200 55 "-" "Mozilla/5.0 (Macintos
+h; U; Intel Mac OS X; en-us) AppleWebKit/523.10.6 (KHTML, like Gecko)
+" "67.60.185.31"';
my $log_pattern = q{(.*) \- \- \[(.*)\] \"(.*) (.*)\?(.*) HTTP\/(.*)\"
+ ([0-9]*) ([0-9]*) \"(.*)\" \"(.*)\" \"(.*)\"};

my @fields = ( $log_line =~ /$log_pattern/ );

print "$_\n" for @fields;
__END__
67.60.185.31
14/Jan/2008:02:25:54 -0800
GET
/display.cgi
2643943|3334115
1.1
200
55
-
Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-us) AppleWebKit/523.10.6
+ (KHTML, like Gecko)
67.60.185.31

However, I'd probably write it this way:

use strict;
use warnings;

my $log_line = '67.60.185.31 - - [14/Jan/2008:02:25:54 -0800] "GET /di
+splay.cgi?2643943|3334115 HTTP/1.1" 200 55 "-" "Mozilla/5.0 (Macintos
+h; U; Intel Mac OS X; en-us) AppleWebKit/523.10.6 (KHTML, like Gecko)
+" "67.60.185.31"';

my $ip_address = qr{ \d{1,3} (?: \. \d{1,3} ){3} }xms;
my $log_pattern
    = qr{
         ( $ip_address )
         \s \S+           # user name
         \s \S+           # user group?
         \s
         \[
         (
         \d\d /           # day
         (?: Jan | Feb | Mar | Apr | May | Jun
           | Jul | Aug | Sep | Oct | Nov | Dec )   # month
          / \d{4}   # year
          : \d\d : \d\d : \d\d    # time
          \s+ \S+                 # timezone
          )
          \]

         \s
         \"
         ( [A-Z]+ )               # method (GET, POST)
         \s+
         ( \S+ ) \? ( \S+ )       # URL parts
         \s+
         HTTP/( 1\.\d )           # protocol version
         \"
         \s

         ( \d+ )                  # response code
         \s+
         ( \d+ )                  # bytes of response

         \s
         \" ( .* ) \"             # referrer
         \s
         \" ( .* ) \"             # user agent

         \s+
         \" ( $ip_address ) \"
     }xms;

my @fields = ( $log_line =~ /$log_pattern/ );

print "$_\n" for @fields;

Having written all that, now I'm betting there's a CPAN module that does this and more.

[gone2015]

Once you have a regex that works, I would urge you to add a little bit of code to watch out for lines that the regex does not cope with. If you've not catered for a rare form of line, or new forms of line are invented in the future, it's better to be told about them -- rather than either silently ignoring them or quietly creating rubbish entries in the data base.