Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

parrallel web requests using http::async

by Anonymous Monk
on Jun 13, 2013 at 14:01 UTC ( #1038746=perlquestion: print w/replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I have a few problems that I hope you might be able to help with. I'm trying to get pull a set of data from a number of webservers about every 10 seconds. Each set contains a url of data, and two urls of images. I need to save each set either in a directory based on the name or a file based on the name.

My problems are:
1. I can't figure out how to match the returned data with the name
2. I'm unable to get this working using basic authentication.
This is what I have so far, but I'm stumped and need help...

#!/usr/bin/perl use strict; use warnings; require HTTP::Request; use Data::Dumper; use HTTP::Async; #load the list of targets from a file #file format is: #server_name,username,password,https://{url} my $file='conf.cfg'; my @data; open(my $fh, '<', $file) or die "Can't read config file '$file' [$!]\n +"; while (my $line = <$fh>) { chomp $line; my @fields = split(/,/, $line); push @data, \@fields; } my $async = HTTP::Async->new; $async->proxy_host(""); $async->proxy_port("8080"); $async->timeout(10); $async->max_request_time(10); $async->slots(100); while(1){ foreach(@data) { my $page; my $name=$_->[0]; my $account=$_->[1]; my $password=$_->[2]; my $url=$_->[3] . "/info.html"; my $port=$_->[3]; $port=~s/.*://g; my $method=$_->[3]; $method=~s/:\/\/.*//g; my $img1=$_->[3]."/image1.jpg"; my $img2=$_->[3]."/image2.jpg"; my $url=$method . "://" . $addr . ":" . $port . "/status/captures" +; my $now = time; $async->add( HTTP::Request->new( GET => $url) ); $async->add( HTTP::Request->new( GET => $img1) ); $async->add( HTTP::Request->new( GET => $img2) ); while ( my $response = $async->wait_for_next_response ) { my $url = sprintf "%-20s,%-25s", $venue, $url; my $len = sprintf "%8s", length $response->content; my $et = sprintf "%5.3f", time - $now; print "$url has length $len and loaded in $et s \n"; if ($response->content eq "Timed out"){ $page="Timed out"; } else{ $page=$response->content; } open (FILE, '>', $name); print FILE $page; close (FILE); } } sleep(10); } exit(0);

Replies are listed 'Best First'.
Re: parrallel web requests using http::async
by BrowserUk (Pope) on Jun 13, 2013 at 14:19 UTC

    When you do the ->add(), it returns a unique id which you can save.

    When you do the ->wait_for_next_response() it will also returns that unique id, which you can then match to the one you saved.

    See the pod for more details.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1038746]
Approved by Happy-the-monk
Front-paged by Corion
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (7)
As of 2018-06-24 22:47 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (126 votes). Check out past polls.