Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

Hi Monks,

I have a few problems that I hope you might be able to help with. I'm trying to get pull a set of data from a number of webservers about every 10 seconds. Each set contains a url of data, and two urls of images. I need to save each set either in a directory based on the name or a file based on the name.

My problems are:
1. I can't figure out how to match the returned data with the name
2. I'm unable to get this working using basic authentication.
This is what I have so far, but I'm stumped and need help...

#!/usr/bin/perl use strict; use warnings; require HTTP::Request; use Data::Dumper; use HTTP::Async; #load the list of targets from a file #file format is: #server_name,username,password,https://{url} my $file='conf.cfg'; my @data; open(my $fh, '<', $file) or die "Can't read config file '$file' [$!]\n +"; while (my $line = <$fh>) { chomp $line; my @fields = split(/,/, $line); push @data, \@fields; } my $async = HTTP::Async->new; $async->proxy_host(""); $async->proxy_port("8080"); $async->timeout(10); $async->max_request_time(10); $async->slots(100); while(1){ foreach(@data) { my $page; my $name=$_->[0]; my $account=$_->[1]; my $password=$_->[2]; my $url=$_->[3] . "/info.html"; my $port=$_->[3]; $port=~s/.*://g; my $method=$_->[3]; $method=~s/:\/\/.*//g; my $img1=$_->[3]."/image1.jpg"; my $img2=$_->[3]."/image2.jpg"; my $url=$method . "://" . $addr . ":" . $port . "/status/captures" +; my $now = time; $async->add( HTTP::Request->new( GET => $url) ); $async->add( HTTP::Request->new( GET => $img1) ); $async->add( HTTP::Request->new( GET => $img2) ); while ( my $response = $async->wait_for_next_response ) { my $url = sprintf "%-20s,%-25s", $venue, $url; my $len = sprintf "%8s", length $response->content; my $et = sprintf "%5.3f", time - $now; print "$url has length $len and loaded in $et s \n"; if ($response->content eq "Timed out"){ $page="Timed out"; } else{ $page=$response->content; } open (FILE, '>', $name); print FILE $page; close (FILE); } } sleep(10); } exit(0);

In reply to parrallel web requests using http::async by Anonymous Monk

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others lurking in the Monastery: (5)
    As of 2018-06-20 04:44 GMT
    Find Nodes?
      Voting Booth?
      Should cpanminus be part of the standard Perl release?

      Results (116 votes). Check out past polls.