Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

parrallel web requests using http::async

by Anonymous Monk
on Jun 13, 2013 at 14:01 UTC ( #1038746=perlquestion: print w/replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I have a few problems that I hope you might be able to help with. I'm trying to get pull a set of data from a number of webservers about every 10 seconds. Each set contains a url of data, and two urls of images. I need to save each set either in a directory based on the name or a file based on the name.

My problems are:
1. I can't figure out how to match the returned data with the name
2. I'm unable to get this working using basic authentication.
This is what I have so far, but I'm stumped and need help...

#!/usr/bin/perl use strict; use warnings; require HTTP::Request; use Data::Dumper; use HTTP::Async; #load the list of targets from a file #file format is: #server_name,username,password,https://{url} my $file='conf.cfg'; my @data; open(my $fh, '<', $file) or die "Can't read config file '$file' [$!]\n +"; while (my $line = <$fh>) { chomp $line; my @fields = split(/,/, $line); push @data, \@fields; } my $async = HTTP::Async->new; $async->proxy_host(""); $async->proxy_port("8080"); $async->timeout(10); $async->max_request_time(10); $async->slots(100); while(1){ foreach(@data) { my $page; my $name=$_->[0]; my $account=$_->[1]; my $password=$_->[2]; my $url=$_->[3] . "/info.html"; my $port=$_->[3]; $port=~s/.*://g; my $method=$_->[3]; $method=~s/:\/\/.*//g; my $img1=$_->[3]."/image1.jpg"; my $img2=$_->[3]."/image2.jpg"; my $url=$method . "://" . $addr . ":" . $port . "/status/captures" +; my $now = time; $async->add( HTTP::Request->new( GET => $url) ); $async->add( HTTP::Request->new( GET => $img1) ); $async->add( HTTP::Request->new( GET => $img2) ); while ( my $response = $async->wait_for_next_response ) { my $url = sprintf "%-20s,%-25s", $venue, $url; my $len = sprintf "%8s", length $response->content; my $et = sprintf "%5.3f", time - $now; print "$url has length $len and loaded in $et s \n"; if ($response->content eq "Timed out"){ $page="Timed out"; } else{ $page=$response->content; } open (FILE, '>', $name); print FILE $page; close (FILE); } } sleep(10); } exit(0);

Replies are listed 'Best First'.
Re: parrallel web requests using http::async
by BrowserUk (Pope) on Jun 13, 2013 at 14:19 UTC

    When you do the ->add(), it returns a unique id which you can save.

    When you do the ->wait_for_next_response() it will also returns that unique id, which you can then match to the one you saved.

    See the pod for more details.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1038746]
Approved by Happy-the-monk
Front-paged by Corion
[stevieb]: can anyone recommend a good IDE for C programming that has *proper* vim plugin support? I've tried Eclipse, Netbeans etc and just can't get anything to work properly on Linux. I'm open to paying, but not on some monthly/yearly plan (just a one-off cost)
[stevieb]: I'd go with CLion as it's very similar to what I use already (intelliJ for Perl, Pycharm for Python), but it's a recurring fee every year

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (7)
As of 2017-01-18 19:12 GMT
Find Nodes?
    Voting Booth?
    Do you watch meteor showers?

    Results (163 votes). Check out past polls.