Re: Fast fetching of HTML response code

[Corion]

I would look at AnyEvent::HTTP, which allows you to run lots (and lots) of HTTP requests in parallel. To reduce bandwidth on errors, you could first test with a HEAD request and follow up with a GET request on success. If you expect most URLs to be successfull and want the body then, I would just use a GET request.

For grouping all your requests, have a look at the ->begin method of CondVars in AnyEvent.

A rough, untested program could look something like this:

#!perl -w
use strict;
use AnyEvent;
use AnyEvent::HTTP;

my $done = AnyEvent->condvar;
while (my $url = <DATA>) {
    $url =~ s!\s+$!!;
    $done->begin( sub { $_[0]->send } );
    http_get $url, sub {
        my ($body, $headers) = @_;
        print "Retrieved $url ($headers->{Status})";
        $done->end
    };
};
print "All requests sent. Waiting for responses.\n";
$done->recv;

print "Done.\n";

__DATA__
http://localhost
http://example.com
http://google.in

Update: Remove whitespace at end of the URLs

[mrguy123]

Hi Corion, thanks for the tip
I am currently using Parallel::ForkManager to make the process faster but will also take a look at AnyEvent::HTTP
The HEAD option, however, does not seem reduce to the time of response so I am looking for new ideas
Guy

[Corion]

You likely can't reduce the time per URL, but you can reduce the total time by making more requests in parallel. Or you can try to find out whether you are bandwidth-limited, or CPU-limited or whether the remote side is slow to respond. My guess is that the remote side is slow to respond.