Using LWP: is there a way to get the server response time?

[isync]

Hi!

Using the LWP set of mudules, especially LWP::UserAgent I am trying to find out how I am able to measure the time between issuing of a request against a server and the start of the server responding.

So far I only managed to measure the time betwen start of the request and finish of the request. But this takes downloading data into account as well. Is there a hidden var or a method to measure just the "performance" of the called server? Maybe by using the :content_cf callback??

[Tomte]

If raw performance (time from "request-came-in" to "about-to-start-sending") is what you want to measure, in my humble opinion you should measure that on the server itself, not the client, otherwise the network will allways interfere with your measurements - in our current projects we send an X-header holding the processing time: X-Time: 0.017s

[Anonymous Monk]

Mmmh. So is a http response alsways a combination of header+response and I have no handle to get into the data-chunks (and their timing) received by LWP::UserAgent?

I am not able to get on the server side. So my only chance is to calculate: ((overall-response-time / transferred-bytes) / my-network-throughput-capacity) or something like that to get a hint of the time used to prepare the response on the server side?

[PreferredUserName]

Libcurl keeps good information on this kind of thing (see CURLINFO_STARTTRANSFER_TIME). Info: http://curl.haxx.se/libcurl/c/curl_easy_getinfo.html

There's a Perl module: WWW::Curl at http://curl.haxx.se/libcurl/perl/

You could also just record the time at intervals in LWP::Protocol::http::request().

[Anonymous Monk]

I am not so familar with the LWP internals and never came to the "LWP::Protocol" branch... How would I poll LWP::Protocol::http::request()? By getting the private $sel_timeout var? A code hint would be great!

I don't want to switch over to curl...

[PreferredUserName]

OK, do a:

   locate Protocol/http.pm

or similar to find where your LWP::Protocol::http module lives. Mine is in /usr/share/perl5/LWP/Protocol/http.pm. Search for the request() subroutine. Add these lines up at the top of it:

   use Time::HiRes qw(time);
   my @times;

Add a line containing "push @times, time();" before the following three lines:

   if ($has_content) { block, 
   ...
   $response = $self->collect($arg, $response, sub {
   ...
   unless ($drop_connection) {

and then add this at the bottom:

   warn "wrote-request    read-headers     read-content\n@times\n";

If this is anything but a quick hack, it might be better to subclass or otherwise hook into LWP::Protocol::http to make it more maintainable. Another approach would be to just do a HEAD and then a GET, with the difference in times being the content-transfer time.