Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Re: check multiple URLS in the same time

by zentara (Archbishop)
on Aug 19, 2012 at 09:36 UTC ( #988313=note: print w/replies, xml ) Need Help??

in reply to check multiple URLS in the same time

Hi, here is an untested piece of code( not written by me ) to show you how to use Parallel::ForkManager to run things concurrently. You could also use threads, but forking is simpler. You could also google for "Parallel ForkManager LWP" and get links like parallel web get with lwp::simple
#!/usr/bin/perl use Parallel::ForkManager; use LWP::Simple; use LWP::UserAgent ; use HTTP::Status ; use HTTP::Request ; %urls = ('drudge'=> '', 'rush' =>'', 'yahoo' => '', 'cds' => '',); foreach $myURL (sort(values(%urls))){ $count++; print "Count is $count\n"; $document = DOCUMENT_RETRIEVER($myURL); } sub DOCUMENT_RETRIEVER{ $myURL=$_[0]; $mit = $myURL; print "Commencing DOCUMENT_RETRIEVER number $iteration for $mit\n"; print "Iteration is $iteration and Count is $count\n"; for ($iteration = $count; $iteration <= $count;$iteration++){ $name = $iteration; print "NAME $name\n" ; my $pm=new Parallel::ForkManager(30); $pm->start and next; print "Starting Child Process $iteration for $mit\n" ; $ua = LWP::UserAgent->new; $ua->agent("$0/0.1 " . $ua->agent); $req = new HTTP::Request 'GET' => "$mit"; $res = $ua->request($req, "$name.html"); print "Process $iteration Complete\n" ; $pm->finish; $pm->wait_all_childs; print "Waiting on children\n"; } undef $name; }

I'm not really a human, but I play one on earth.
Old Perl Programmer Haiku ................... flash japh

Replies are listed 'Best First'.
Re^2: check multiple URLS in the same time
by Anonymous Monk on Aug 19, 2012 at 10:24 UTC

    You could also use threads, but forking is simpler.

    Absolutely, its why use Parallel::ForkManager instead of forks :)

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://988313]
[marto]: good morning all

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (7)
As of 2017-06-26 09:33 GMT
Find Nodes?
    Voting Booth?
    How many monitors do you use while coding?

    Results (575 votes). Check out past polls.