Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

parrallel web requests using http::async

by Anonymous Monk
on Jun 13, 2013 at 14:01 UTC ( #1038746=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I have a few problems that I hope you might be able to help with. I'm trying to get pull a set of data from a number of webservers about every 10 seconds. Each set contains a url of data, and two urls of images. I need to save each set either in a directory based on the name or a file based on the name.

My problems are:
1. I can't figure out how to match the returned data with the name
2. I'm unable to get this working using basic authentication.
This is what I have so far, but I'm stumped and need help...

#!/usr/bin/perl use strict; use warnings; require HTTP::Request; use Data::Dumper; use HTTP::Async; #load the list of targets from a file #file format is: #server_name,username,password,https://{url} my $file='conf.cfg'; my @data; open(my $fh, '<', $file) or die "Can't read config file '$file' [$!]\n +"; while (my $line = <$fh>) { chomp $line; my @fields = split(/,/, $line); push @data, \@fields; } my $async = HTTP::Async->new; $async->proxy_host("proxy.my.company.com"); $async->proxy_port("8080"); $async->timeout(10); $async->max_request_time(10); $async->slots(100); while(1){ foreach(@data) { my $page; my $name=$_->[0]; my $account=$_->[1]; my $password=$_->[2]; my $url=$_->[3] . "/info.html"; my $port=$_->[3]; $port=~s/.*://g; my $method=$_->[3]; $method=~s/:\/\/.*//g; my $img1=$_->[3]."/image1.jpg"; my $img2=$_->[3]."/image2.jpg"; my $url=$method . "://" . $addr . ":" . $port . "/status/captures" +; my $now = time; $async->add( HTTP::Request->new( GET => $url) ); $async->add( HTTP::Request->new( GET => $img1) ); $async->add( HTTP::Request->new( GET => $img2) ); while ( my $response = $async->wait_for_next_response ) { my $url = sprintf "%-20s,%-25s", $venue, $url; my $len = sprintf "%8s", length $response->content; my $et = sprintf "%5.3f", time - $now; print "$url has length $len and loaded in $et s \n"; if ($response->content eq "Timed out"){ $page="Timed out"; } else{ $page=$response->content; } open (FILE, '>', $name); print FILE $page; close (FILE); } } sleep(10); } exit(0);

Comment on parrallel web requests using http::async
Download Code
Re: parrallel web requests using http::async
by BrowserUk (Pope) on Jun 13, 2013 at 14:19 UTC

    When you do the ->add(), it returns a unique id which you can save.

    When you do the ->wait_for_next_response() it will also returns that unique id, which you can then match to the one you saved.

    See the pod for more details.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1038746]
Approved by Happy-the-monk
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (9)
As of 2014-08-29 23:22 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (289 votes), past polls