Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Re: check multiple URLS in the same time

by zentara (Archbishop)
on Aug 19, 2012 at 09:36 UTC ( #988313=note: print w/replies, xml ) Need Help??

in reply to check multiple URLS in the same time

Hi, here is an untested piece of code( not written by me ) to show you how to use Parallel::ForkManager to run things concurrently. You could also use threads, but forking is simpler. You could also google for "Parallel ForkManager LWP" and get links like parallel web get with lwp::simple
#!/usr/bin/perl use Parallel::ForkManager; use LWP::Simple; use LWP::UserAgent ; use HTTP::Status ; use HTTP::Request ; %urls = ('drudge'=> '', 'rush' =>'', 'yahoo' => '', 'cds' => '',); foreach $myURL (sort(values(%urls))){ $count++; print "Count is $count\n"; $document = DOCUMENT_RETRIEVER($myURL); } sub DOCUMENT_RETRIEVER{ $myURL=$_[0]; $mit = $myURL; print "Commencing DOCUMENT_RETRIEVER number $iteration for $mit\n"; print "Iteration is $iteration and Count is $count\n"; for ($iteration = $count; $iteration <= $count;$iteration++){ $name = $iteration; print "NAME $name\n" ; my $pm=new Parallel::ForkManager(30); $pm->start and next; print "Starting Child Process $iteration for $mit\n" ; $ua = LWP::UserAgent->new; $ua->agent("$0/0.1 " . $ua->agent); $req = new HTTP::Request 'GET' => "$mit"; $res = $ua->request($req, "$name.html"); print "Process $iteration Complete\n" ; $pm->finish; $pm->wait_all_childs; print "Waiting on children\n"; } undef $name; }

I'm not really a human, but I play one on earth.
Old Perl Programmer Haiku ................... flash japh

Replies are listed 'Best First'.
Re^2: check multiple URLS in the same time
by Anonymous Monk on Aug 19, 2012 at 10:24 UTC

    You could also use threads, but forking is simpler.

    Absolutely, its why use Parallel::ForkManager instead of forks :)

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://988313]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (10)
As of 2017-10-19 16:47 GMT
Find Nodes?
    Voting Booth?
    My fridge is mostly full of:

    Results (255 votes). Check out past polls.