Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Using mod_log_post output to unit test Apache httpd conf changes.

by jffry (Hermit)
on Jan 05, 2010 at 19:14 UTC ( #815803=CUFP: print w/replies, xml ) Need Help??

I needed to capture the data from HTTP POST requests on our Apache httpd server. I used mod_log_post. I needed this data so I could to use it as a basis for a quickie sniff test against changes to our Apache httpd server I was planning.

I wrote a script that parsed the post_log output created by mod_log_post and perform the exact same HTTP POST request on whatever host I pointed it at.

This is nothing fancy. Just a quick hack to get a unit test going. For example, if I wanted a real stressor, I should probably make it multi-threaded, and so on.

WARNING:You should read my comments on how I decide what is POST data and what is not. I don't know the HTTP spec well enough, and your POST data might not obey these rules.

Here is the whole thing, screen scraped right from my console.

#!/usr/bin/perl -w # ** Design Note ** # # Right now this script is data-driven. Thus, I could have used the ' +-an' # flags to Perl. This turns Perl into an awk-style script. However, +I may # want to make this non-data-driven later, so it isn't worth it. use strict; use warnings; # Need these 3 for do_test() that performs the actual HTTP request. use HTTP::Request; use HTTP::Headers; use LWP::UserAgent; # Process command line options/flags. use Getopt::Std; my %opt; getopts('h:pv', \%opt); # Debug logging on or off. my $verbose = 0; if (defined $opt{'v'}) { $verbose = 1; } # Pretend on or off. my $pretend = 0; if (defined $opt{'p'}) { $pretend = 1; } # Which host to test. # This is a global. It doesn't have to be. Might refactor later, but + now it is OK. my $host = 'localhost:80'; if (defined $opt{'h'}) { $host = $opt{'h'}; } my $flag = 0; # When true, current line is POST data. my $uri; # Holds URI string. my $data; # Holds POST data for a single URI. while (<>) { # If the line starts with "POST" this must be the URI line. if (/^POST/) { print STDERR 'DEBUG: /^POST/ re line = ' . $_ . "\n" i +f $verbose; $uri = (split /\s+/, $_)[1]; print "$uri\n\n" if $pretend; next; } # A number by itself signals the beginning of the HTTP body, w +hich contains the POST data. if (/^\d+$/ && $flag == 0) { print STDERR 'DEBUG: /^\d+$/ re turned flag on' . "\n" + if $verbose; $flag = 1; next; } if ($flag) { $data .= $_; print if $pretend; } # A blank line by itself signals the end of the POST data. if (/^\s*$/ && $flag == 1) { print STDERR 'DEBUG: /^\s*$/ re turned flag off' . "\n +" if $verbose; print "\n" if ($pretend); do_test($uri, $data); $data = ''; $uri = ''; $flag = 0; next; } } sub do_test { my ($uri, $data) = @_; # Uses the $host global. If you're going to refactor and pass + 3 args to # this sub, then please convert to named args (pass a hash ref +). my $head = HTTP::Headers->new( Content_Type => 'txt/xml', ); $uri = 'http://' . $host . $uri; my $req = HTTP::Request->new( 'POST', $uri, $head, $data); if ($pretend) { print STDERR "DEBUG: Pretend flag set. Not really doin +g HTTP request.\n" if $verbose; print $req->as_string() . "\n\n"; } else { my $ua = LWP::UserAgent->new; my $resp; $resp = $ua->request($req); # This actually performs t +he HTTP request. print STDERR $resp->status_line . "\n" if $verbose; } } __END__ =pod =head1 Usage ./ [options] post_log_files ... =head1 Options =head2 -h hostname:port To target a system other than localhost:80. =head2 -p Pretend/print only. Don't test, just print relevant lines of post_log + file. =head2 -v Turns on verbose logging to stderr. =cut

Here is what a segment of post_log output looks like.

==f3731b40============================== Request: bob - - [29/Dec/2009:18:11:59 --0500] "POST /our/uri +/ HTT P/1.0" 200 218 "-" "-" SzqMvwoEVDQAAHGZBgIAAAAB "-" Handler: jakarta-servlet ---------------------------------------- POST /our/uri/ HTTP/1.0 Host: bob:80 Content-Type: text/xml Content-Length: 149 149 <?xml version='1.0' encoding='ISO-8859-1'?><BobSaysSomething><SomeData +Date>12/29/2009</SomeDataDate></BobSaysSomething> HTTP/1.1 200 OK Content-Length: 218 Connection: close Content-Type: text/xml --f3731b40--

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: CUFP [id://815803]
Approved by zwon
Front-paged by Arunbear
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (4)
As of 2017-07-20 23:22 GMT
Find Nodes?
    Voting Booth?
    I came, I saw, I ...

    Results (316 votes). Check out past polls.