Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling

Live re(p)laying of HTTP requests

by liz (Monsignor)
on Jun 25, 2004 at 19:07 UTC ( #369720=perlquestion: print w/replies, xml ) Need Help??
liz has asked for the wisdom of the Perl Monks concerning the following question:

I'm looking for an easy way to re(p)lay HTTP requests, sent to one (live) server, immediately (or later) to another (test) server, for both GET and POST requests. With as little interference as possible on the original (live) server.

The way I figure it, this would ideally be a module that would install itself as an Apache PerlHandler in a mod_perl environment. This module would take the request object and write out the input headers, method, URL etc. to a log file or syslog daemon in some format.

Another (part of the) module would be capable of reading the log file and sending an identical request to a (test) server of choice. This could be either a direct relay (when reading the log file with a tail -f) or a delayed replay (whenever you want to process the log file).

I've looked at HTTP::Recorder and WWW::Mechanize, but they seem to be overkill (and I don't want to add the extra complexity of a proxy server).

Is there such a beast already? If not, would it make sense to make such a beast?


Replies are listed 'Best First'.
Re: Live re(p)laying of HTTP requests
by jeffa (Bishop) on Jun 25, 2004 at 20:16 UTC

    Just a random thought ... but have you seen Firefox's Live HTTP Headers extension? I am not sure how easy it would be to extend this or automate it, but if you are just needing to inspect the headers "by hand", then this is a valuable tool.


    (the triplet paradiddle with high-hat)
      Headers are not the problem. POST parameters are. And it needs to run (semi) automatically.

      But I agree the Live HTTP Headers plugin is a good thing to have as a developer. Thanks for the reminder.


Re: Live re(p)laying of HTTP requests
by perrin (Chancellor) on Jun 25, 2004 at 19:15 UTC
    Here's a way to do it with httperf. (Warning: PDF file)
      I'm not interested in benchmarking. This is really about testing new releases of (legacy) software that work fine when run by themselves, but which blow up for some reason in a mod_perl environment.

      I don't see httperf allowing me to do what I want to do. It has some support for replaying sessions, but they don't seem to support POST requests.

      Am I missing something?


        I thought I remembered it supporting POST, but I'm not psitive. What I've done with it is turn an apache log into a httperf session and play it back. If you really need it to be live, I would look at LWP instead. There are a couple of modules for proxying requests with LWP, which you can find on CPAN, but you'd have to do a bit of work to make them actually run the request locally as well as proxy it. Ideally, I would put the proxy stuff in a cleanup handler.
Re: Live re(p)laying of HTTP requests
by gaal (Parson) on Jun 26, 2004 at 12:47 UTC

    webspy from the dsniff package picks HTTP GET and POST requests from the wire, and massages them into browser remote commands to place from your own browser. Not quite what you're looking for (it's even in c), but this avoids the proxy, at least from the original user's perspective.

    So what I'd do is try to embed the core logic of webspy in a perl script with Inline, and rather than repeat the request immediately in the original netscape remote-ish way, massage the data into a log file (changing the client IP(/hostname?) along the way to some parameterized value).

    I'd suggest doing the whole thing in Perl really, but unfortunately libnids doesn't have perl bindings. Hmmm.

Re: Live re(p)laying of HTTP requests
by Mercio (Scribe) on Jun 25, 2004 at 19:32 UTC
    I don't know if I understand what you are wanting, but i think I do. Try using LWP. You can find it on PPM. It's a very nice module that can recieve and send http request when the script is activated. Or maybe I'm just a total idiot..?
      Indeed. LWP::UserAgent would probably be used, although I'm also considering using IO::Socket directly.

      But I was wondering whether someone would have made the things I need around to LWP::UserAgent already.

      Or whether there would be a better way to achieve what I want: re(p)laying requests, received on a live server, on another (test) server. And without needing any proxy setup.


Re: Live re(p)laying of HTTP requests
by andyf (Pilgrim) on Jun 26, 2004 at 13:28 UTC
    Just to reinforce Gaals suggestion, if this is just for analysis and test why not get down and dirty with the actual packets on the wire, ethereal is an alternative to dsniff, tail -f output and write a little filter to spot your POSTS.
Re: Live re(p)laying of HTTP requests
by knoebi (Friar) on Jun 27, 2004 at 14:49 UTC
    Update: I re-read your question and unfortunaly my answer invokes a (slow) proxy server, which is not a good solution in a live enviroment (it's only a solution for a test enviroment).

    Hi Liz,

    i was looking for exactly the same thing as you some time ago. Even if you write you don't want to add the extra complexity of a proxy server, i suggest you give it a try because this is an extremly easy way of creating (recording) HTTP requests. for playback, you don't need a proxy server anyway.

    My solution is based on the POE-Web Proxyserver from the POE Cookbook. It just writes all the Request with timestamps to a MySQL DB (via Class::DBI).

    #!/usr/bin/perl use warnings; use strict; use POE; use POE::Component::Server::TCP; use POE::Component::Client::HTTP; use POE::Filter::HTTPD; use HTTP::Response; use Data::Dumper; use IO::All; use DB::DBI; use Time::HiRes; # use CGI; use CGI::Session; sub DUMP_REQUESTS () { 0 } sub DUMP_RESPONSES () { 0 } sub LISTEN_PORT () { 8088 } ### Spawn a web client to fetch requests through. POE::Component::Client::HTTP->spawn( Alias => 'ua' ); ### Spawn a web server. # The ClientInput function is called to deal with client input. # ClientInput's callback function will receive entire HTTP requests # because this server uses POE::Filter::HTTPD to parse its input. # # InlineStates let us attach our own events and handlers to a TCP # server. Here we attach a handler for the got_response event, which # will be sent to us by Client::HTTP when it has fetched something. POE::Component::Server::TCP->new ( Alias => "web_server", Port => LISTEN_PORT, ClientFilter => 'POE::Filter::HTTPD', ClientInput => \&handle_http_request, InlineStates => { got_response => \&handle_http_response, }, ); ### Run the proxy until it is done, then exit. print "http proxy started on localhost:".&LISTEN_PORT." [OK]\n"; POE::Kernel->run(); exit 0; ### Handle HTTP requests from the client. Pass them to the HTTP ### client component for further processing. Optionally dump the ### request as text to STDOUT. sub handle_http_request { my ( $kernel, $heap, $request ) = @_[ KERNEL, HEAP, ARG0 ]; # print Data::Dumper::Dumper ($request); # If the request is really a HTTP::Response, then it indicates a # problem parsing the client's request. Send the response back so # the client knows what's happened. if ( $request->isa("HTTP::Response") ) { $heap->{client}->put($request); $kernel->yield("shutdown"); return; } my $tmp_str=$request->as_string(); #print $tmp_str; $tmp_str=~/CARMEN_SID=(\w+)/; $request->{_sid}=$1; $request->{_cgi_session} = new CGI::Session(undef, $request->{_sid +}, {Directory=>'/tmp'}); $request->{_sid} = $request->{_cgi_session}->id; $request->{Time_HiRes_gettimeofday} = [Time::HiRes::gettimeofday]; $request->{Time_HiRes_time} = Time::HiRes::time(); # Client::HTTP doesn't support keep-alives yet. $request->header( "Connection", "close" ); $request->header( "Proxy-Connection", "close" ); $request->remove_header("Keep-Alive"); $kernel->post( "ua" => "request", "got_response", $request ); } ### Handle HTTP responses from the POE::Component::Client::HTTP we've ### spawned at the beginning of the program. Send each response back ### to the client that requested it. Optionally display the response ### as text. sub handle_http_response { my ( $kernel, $heap ) = @_[ KERNEL, HEAP ]; my $http_response = $_[ARG1]->[0]; $http_response->push_header( Set_Cookie => 'CARMEN_SID='.$http_res +ponse->{_request}->{_sid} ); my $response_type = $http_response->content_type(); if ( $response_type =~ /^text/i ) { $http_response->as_string() if DUMP_RESPONSES; } else { print "Response wasn't text.\n" if DUMP_RESPONSES; } my $obj_http_response = DB::http_response->create({ http_response => Data::Dumper::Dumper ($http_response) +, time_request => $http_response->{_request}->{Time_HiRe +s_time}, time_http_response => Time::HiRes::time(), time_interval => Time::HiRes::tv_interval ( $http_resp +onse->{_request}->{Time_HiRes_gettimeofday}, [Time::HiRes::gettimeofd +ay]), sid => $http_response->{_request}->{_sid}, uri => $http_response->request->uri }); $obj_http_response->update(); # Avoid sending the response if the client has gone away. $heap->{client}->put($http_response) if defined $heap->{client}; # Shut down the client's connection when the response is sent. $kernel->yield("shutdown"); }
    replaying the requests is really easy, this script will do it:
    #!/usr/bin/perl use warnings; use strict; use Data::Dumper; use IO::All; use LWP::UserAgent; use URI::http; use DB::DBI; my @objs = DB::http_response->retrieve_all; foreach my $obj (@objs) { my $VAR1; eval ($obj->get('http_response')); my $ua = LWP::UserAgent->new; my $response = $ua->request($VAR1->{_request}); print $VAR1->{_request}->uri."\n"; }
    just in case you're interested in DB::DBI (bad name, i know):
    #!/usr/bin/perl use warnings; use strict; package DB::DBI; use base 'Class::DBI'; __PACKAGE__->connection('dbi:mysql:carmen', 'root', ''); package DB::http_response; use base 'DB::DBI'; __PACKAGE__->table('http_response'); __PACKAGE__->columns(All =>qw/http_response_id http_response time_request time_http_response time_interval sid uri /); package DB::test_run; use base 'DB::DBI'; __PACKAGE__->table('test_run'); __PACKAGE__->columns(All =>qw/test_run_id name /);
    for me this works extremly well. all our testing by hand is recorded since we started testing over this proxy server. and because its all in perl its very easy to change/add all kind of things, like supporting dynamic ID generation, SID, and so on.



Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://369720]
Approved by Sandy
Front-paged by grinder
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (5)
As of 2018-10-21 18:59 GMT
Find Nodes?
    Voting Booth?
    When I need money for a bigger acquisition, I usually ...

    Results (119 votes). Check out past polls.