http://www.perlmonks.org?node_id=617553


in reply to running multiple LWP gets

First search http://groups.google.com for "LWP Parallel::ForkManager"

I don't see why LWP::Parallel wasn't showing a speed improvement unless you set the max_req too low. $ua->max_req(30) defaults to 5.

I don't think threads will help you, except to give realtime progress reports for each download. Try this: (there are similar examples on groups.google).

#!/usr/bin/perl use Parallel::ForkManager; use LWP::Simple; use LWP::UserAgent ; use HTTP::Status ; use HTTP::Request ; %urls = ('drudge'=> 'http://www.drudgereport.com', 'rush' =>'http://www.rushlimbaugh.com/home/today.guest.html', 'yahoo' => 'http://www.yahoo.com', 'cds' => 'http://www.cdsllc.com/',); foreach $myURL (sort(values(%urls))){ $count++; print "Count is $count\n"; $document = DOCUMENT_RETRIEVER($myURL); } sub DOCUMENT_RETRIEVER{ $myURL=$_[0]; $mit = $myURL; print "Commencing DOCUMENT_RETRIEVER number $iteration for $mit\n"; print "Iteration is $iteration and Count is $count\n"; for ($iteration = $count; $iteration <= $count;$iteration++){ $name = $iteration; print "NAME $name\n" ; my $pm=new Parallel::ForkManager(30); $pm->start and next; print "Starting Child Process $iteration for $mit\n" ; $ua = LWP::UserAgent->new; $ua->agent("$0/0.1 " . $ua->agent); $req = new HTTP::Request 'GET' => "$mit"; $res = $ua->request($req, "$name.html"); print "Process $iteration Complete\n" ; $pm->finish; $pm->wait_all_childs; print "Waiting on children\n"; } undef $name; }

I'm not really a human, but I play one on earth. Cogito ergo sum a bum

Replies are listed 'Best First'.
Re^2: running multiple LWP gets
by gferguson (Acolyte) on May 25, 2007 at 18:57 UTC
    DOH! I forgot about the newsgroups... yes, there's lots of mention there, so I'll see what I can dig up.

    Thanks!

    PS - thanks for the sample code. I'll poke at it and see what it does.

      Doh! :-), as you poke, just remember, I grabbed it without much testing. Now that I look close, it dosn't look like it sets up the Parallel ForkManager in a way that may not run properly.... but the Parallel ForkManager has some odd syntax since it's an object.

      I'm not really a human, but I play one on earth. Cogito ergo sum a bum