Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

parrallel web requests using http::async

by Anonymous Monk
on Jun 13, 2013 at 14:01 UTC ( #1038746=perlquestion: print w/replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I have a few problems that I hope you might be able to help with. I'm trying to get pull a set of data from a number of webservers about every 10 seconds. Each set contains a url of data, and two urls of images. I need to save each set either in a directory based on the name or a file based on the name.

My problems are:
1. I can't figure out how to match the returned data with the name
2. I'm unable to get this working using basic authentication.
This is what I have so far, but I'm stumped and need help...

#!/usr/bin/perl use strict; use warnings; require HTTP::Request; use Data::Dumper; use HTTP::Async; #load the list of targets from a file #file format is: #server_name,username,password,https://{url} my $file='conf.cfg'; my @data; open(my $fh, '<', $file) or die "Can't read config file '$file' [$!]\n +"; while (my $line = <$fh>) { chomp $line; my @fields = split(/,/, $line); push @data, \@fields; } my $async = HTTP::Async->new; $async->proxy_host(""); $async->proxy_port("8080"); $async->timeout(10); $async->max_request_time(10); $async->slots(100); while(1){ foreach(@data) { my $page; my $name=$_->[0]; my $account=$_->[1]; my $password=$_->[2]; my $url=$_->[3] . "/info.html"; my $port=$_->[3]; $port=~s/.*://g; my $method=$_->[3]; $method=~s/:\/\/.*//g; my $img1=$_->[3]."/image1.jpg"; my $img2=$_->[3]."/image2.jpg"; my $url=$method . "://" . $addr . ":" . $port . "/status/captures" +; my $now = time; $async->add( HTTP::Request->new( GET => $url) ); $async->add( HTTP::Request->new( GET => $img1) ); $async->add( HTTP::Request->new( GET => $img2) ); while ( my $response = $async->wait_for_next_response ) { my $url = sprintf "%-20s,%-25s", $venue, $url; my $len = sprintf "%8s", length $response->content; my $et = sprintf "%5.3f", time - $now; print "$url has length $len and loaded in $et s \n"; if ($response->content eq "Timed out"){ $page="Timed out"; } else{ $page=$response->content; } open (FILE, '>', $name); print FILE $page; close (FILE); } } sleep(10); } exit(0);

Replies are listed 'Best First'.
Re: parrallel web requests using http::async
by BrowserUk (Pope) on Jun 13, 2013 at 14:19 UTC

    When you do the ->add(), it returns a unique id which you can save.

    When you do the ->wait_for_next_response() it will also returns that unique id, which you can then match to the one you saved.

    See the pod for more details.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1038746]
Approved by Happy-the-monk
Front-paged by Corion
[1nickt]: Corion HPs? ugh. I was impressed with Lenovo's gaming laptops; if it weren't for the red backlighting and general flashiness of the aesthetic, I might have gone with that. But all that was until I discovered that the Dell Precision line is still around.
[1nickt]: stevieb I don;t doubt that there's a difference. BestBuy has "consumer" models only on display.
[ambrus]: 1nickt: for some reason, these days they call every computer "gaming", even ones that gamers wouldn't buy. I've bought a keyboard that was labelled "gamer", despite that it has hard springs and seems to be way better for typing than for gaming;
[1nickt]: I though the gamers like that because they bash the keys so hard.
[ambrus]: and I've seen motherboards with no fast expansion ports for a video card but built-in hardware RAID advertized as "gaming".
LanX has a shaming laptop
[ambrus]: 1nickt: my impression is that the gamers like the softer springs, because fast reaction time is more important to them then feedback from keypresses to recognize typos.

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (14)
As of 2017-03-27 13:39 GMT
Find Nodes?
    Voting Booth?
    Should Pluto Get Its Planethood Back?

    Results (320 votes). Check out past polls.