regexp matching bad stuff ...

[raiten]

Hello,

I'm trying to dispatch some contents from an apache log depending on a regexp and i have a problem. It seems strings which don't match my regexp goes in the matched area ...

example command-line:

$ cat access_log | perl -pe 'if (s/^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3
+}).*"(GET|POST|HEAD) (.*?) HTTP\/.*$/$1,$3/) { print } else { print S
+TDERR }' 2>out.err > out.csv

results in STDIN

[...]
72.30.161.243,/
72.30.161.243,/
125.224.206.168 - - [04/Oct/2009:00:13:42 +0200] "-" 408 - "-" "-"
125.224.206.168 - - [04/Oct/2009:00:13:42 +0200] "-" 408 -
125.224.206.168 - - [04/Oct/2009:00:13:47 +0200] "CONNECT 203.188.201.
+253:25 HTTP/1.1" 404 516 "-" "-"
125.224.206.168 - - [04/Oct/2009:00:13:47 +0200] "CONNECT 203.188.201.
+253:25 HTTP/1.1" 404 516
96.243.255.188,//phpMyAdmin/
[...]

STDERR has only valid contents (not matching regexp)

corresponding part of the source file:

72.30.161.243 - - [03/Oct/2009:17:21:43 +0200] "GET / HTTP/1.0" 404 51
+6 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.c
+o
m/help/us/ysearch/slurp)"
72.30.161.243 - - [03/Oct/2009:17:21:43 +0200] "GET / HTTP/1.0" 404 51
+6
125.224.206.168 - - [04/Oct/2009:00:13:42 +0200] "-" 408 - "-" "-"
125.224.206.168 - - [04/Oct/2009:00:13:42 +0200] "-" 408 -
125.224.206.168 - - [04/Oct/2009:00:13:47 +0200] "CONNECT 203.188.201.
+253:25 HTTP/1.1" 404 516 "-" "-"
125.224.206.168 - - [04/Oct/2009:00:13:47 +0200] "CONNECT 203.188.201.
+253:25 HTTP/1.1" 404 516
96.243.255.188 - - [04/Oct/2009:00:26:17 +0200] "GET //phpMyAdmin/ HTT
+P/1.1" 404 516 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
96.243.255.188 - - [04/Oct/2009:00:26:17 +0200] "GET //phpMyAdmin/ HTT
+P/1.1" 404 516

Has someone encounters a similar bug ? or is it my regexp ? seems hard to believe that it matched the CONNECT line ...
Normally, out.csv must contains only csv lines.

thanks
Best regards

[ikegami]

Replace -p (which unconditionally prints every line) with -n (which doesn't)

While you're at it, replace
cat access_log | perl ...
with
perl ... < access_log

[kennethk]

According to perlrun:

-p

causes Perl to assume the following loop around your program, which makes it iterate over filename arguments somewhat like sed:

   1. LINE:
   2. while (<>) {
   3.     ... # your program goes here
   4. } continue {
   5.     print or die "-p destination: $!\n";
   6. }

If a file named by an argument cannot be opened for some reason, Perl warns you about it, and moves on to the next file. Note that the lines are printed automatically. An error occurring during printing is treated as fatal. To suppress printing use the -n switch. A -p overrides a -n switch.

BEGIN and END blocks may be used to capture control before or after the implicit loop, just as in awk.

And so, accordingly, you should be using the -n switch (wraps your script in a loop w/o the print statement) in place of the -p.

[ikegami]

Earlier I recommended

perl -ne'if (s/.../.../) { print } else { print STDERR }'
    <access_log 2>out.err >out.csv

What I had in mind was

perl -ne'print { s/.../.../ ? STDOUT : STDERR } $_'
    <access_log 2>out.err >out.csv

It just occured to me that it can be shortened to

perl -pe'select s/.../.../ ? STDOUT : STDERR'
    <access_log 2>out.err >out.csv

[Fletch]

Not a direct solution, but perhaps consider using something like Logfile::Access or Regexp::Log::Common rather than reinventing this particular wheel.