Hello. Trying to use HTTP::Async to do some concurrent downloading, but it seems to take much longer than a traditional HTTP request made with LWP::UserAgent.
Minimal code below. I do a grab of a URL (I've replaced my internal URLs with a URL from the BBC website). When I run it I get:-
Doing it 1st way
1360837557 doing request
1360837557 done
1360837557 DEBUG: Status is: 200 OK
1360837557 LENGTH: 153421
1360837557 Doing it 2nd way
1360837557 NEWPOLL ADDED: 1
1360837557 DEBUG: Checking...
1360837558 DEBUG: Checking...
1360837559 DEBUG: Checking...
1360837561 DEBUG: Checking...
1360837562 DEBUG: Checking...
1360837563 DEBUG: Checking...
1360837564 DEBUG: Checking...
1360837565 DEBUG: Checking...
1360837566 DEBUG: Checking...
1360837567 DEBUG: Checking...
1360837568 DEBUG: Got response reqid=1
1360837568 DEBUG: Status is: 200 OK
1360837568 LENGTH: 151059
So the first attempt (via LWP) completes in a fraction of a second.
The subsequent request (using HTTP::Async) takes 11 seconds or so.
Looking at the tcpdump of the requests being sent I can't really see any difference between the two. And both are made by the same call to HTTP::Request anyway.
Code follows:-
#!/usr/bin/perl
# Turn of auto flush for stdout
$|=1;
use HTTP::Async;
use HTTP::Request::Common qw(POST);
use LWP::UserAgent;
use strict;
binmode STDOUT, ":utf8";
my $http_async = HTTP::Async->new( poll_interval => 0.05 );
my $url="http://www.bbc.co.uk/sport/0/";
# $url="http://www.google.com/";
print "Doing it 1st way\n";
my $ua=LWP::UserAgent->new;
print time()." doing request\n";
my $resp=$ua->request( HTTP::Request->new( "GET", $url ) );
print time()." done\n";
print time()." DEBUG: Status is: ".$resp->status_line."\n";
print time()." LENGTH: ".length($resp->as_string)."\n";
print time()." Doing it 2nd way\n";
my $reqid = $http_async->add( HTTP::Request->new( "GET", $url ) );
print time()." NEWPOLL ADDED: $reqid\n";
while( 1 ) {
# Check for any results
if( $http_async->empty() ) {
print time()." DEBUG: to_send=".$http_async->to_send_c
+ount." in_progress=".$http_async->in_progress_count." to_return=".$ht
+tp_async->to_return_count." total=".$http_async->total_count."\n";
sleep(1);
next;
}
print time()." DEBUG: Checking...\n";
my ( $resp, $reqid ) = $http_async->wait_for_next_response(1.0
+);
if( !defined( $resp ) ) {
next;
}
print time()." DEBUG: Got response reqid=$reqid\n";
print time()." DEBUG: Status is: ".$resp->status_line."\n";
print time()." LENGTH: ".length($resp->as_string)."\n";
last;
}
Any ideas?
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
Outside of code tags, you may need to use entities for some characters:
| |
For: |
|
Use: |
| & | | & |
| < | | < |
| > | | > |
| [ | | [ |
| ] | | ] |
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.
|
|