Re: Stateful Browsing and link extraction with AnyEvent::HTTP

[BrowserUk]

I had thought about using AnyEvent::DNS but there don't seem to be any obvious ways of getting LWP to use its results rather than doing its own synchronous resolution (via the OS).

Surely, if you resolve the domain name yourself (asynchronously or not), and then supply the resolved dotted decimal as part of the url you supply to LWP, it won't have to, or be able to, do the resolution again itself?

(I know; you don't like being called Shirley:)

[Corion]

I think it's not that simple because LWP needs to generate the appropriate Host: header. But if you generate that header in your code, I think you should be able to reuse most of what LWP already supplies.

Alternatively, maybe you can even supply the appropriate opened (blocking) socket to LWP after having made an (asynchronous) connection to the host. But that involves source diving, surely.

[james2vegas]

The web server may have multiple named virtual servers on the same IP, in which case it will check the Host passed in with the request to determine what webroot to use

[BrowserUk]

Just supply the Host: header yourself, you have the appropriate information.

[Anonymous Monk]

It will work for non-authenticated requests easy, but not for authenticated ones
Read more... (3 kB)

[BrowserUk]

This seems to work ok? (Note the extra line!):

#!/usr/bin/perl --

use strict;
use warnings;

use WWW::Mechanize 1.54;

my $ua = WWW::Mechanize->new;
$ua->add_handler(
    "request_send",  sub {
        $_[0]->content('');
        $_[0]->dump;
        return
    }
);

$ua->add_handler(
    "response_done", sub {
        $_[0]->content('');
        $_[0]->dump;
        return
    }
);

my $uri = URI->new('http://jigsaw.w3.org/HTTP/Basic/');
$ua->credentials( $uri->host_port, qw' test guest guest ');

$ua->get( $uri );

print "\n--- SPECIFY HOST MANUALLY ---\n";

my $host = $uri->host( $ua->res->header('Client-peer') );
$ua->credentials( $uri->host_port, qw' test guest guest '); ### ADDED 
+##
$ua->get( $uri, Host => $host );

__END__
GET http://jigsaw.w3.org/HTTP/Basic/
Accept-Encoding: gzip
User-Agent: WWW-Mechanize/1.66

[Anonymous Monk]

Why yes it does, pure genius :D