Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

Re: running multiple LWP gets

by zentara (Archbishop)
on May 25, 2007 at 17:34 UTC ( #617553=note: print w/replies, xml ) Need Help??

in reply to running multiple LWP gets

First search for "LWP Parallel::ForkManager"

I don't see why LWP::Parallel wasn't showing a speed improvement unless you set the max_req too low. $ua->max_req(30) defaults to 5.

I don't think threads will help you, except to give realtime progress reports for each download. Try this: (there are similar examples on

#!/usr/bin/perl use Parallel::ForkManager; use LWP::Simple; use LWP::UserAgent ; use HTTP::Status ; use HTTP::Request ; %urls = ('drudge'=> '', 'rush' =>'', 'yahoo' => '', 'cds' => '',); foreach $myURL (sort(values(%urls))){ $count++; print "Count is $count\n"; $document = DOCUMENT_RETRIEVER($myURL); } sub DOCUMENT_RETRIEVER{ $myURL=$_[0]; $mit = $myURL; print "Commencing DOCUMENT_RETRIEVER number $iteration for $mit\n"; print "Iteration is $iteration and Count is $count\n"; for ($iteration = $count; $iteration <= $count;$iteration++){ $name = $iteration; print "NAME $name\n" ; my $pm=new Parallel::ForkManager(30); $pm->start and next; print "Starting Child Process $iteration for $mit\n" ; $ua = LWP::UserAgent->new; $ua->agent("$0/0.1 " . $ua->agent); $req = new HTTP::Request 'GET' => "$mit"; $res = $ua->request($req, "$name.html"); print "Process $iteration Complete\n" ; $pm->finish; $pm->wait_all_childs; print "Waiting on children\n"; } undef $name; }

I'm not really a human, but I play one on earth. Cogito ergo sum a bum

Replies are listed 'Best First'.
Re^2: running multiple LWP gets
by gferguson (Acolyte) on May 25, 2007 at 18:57 UTC
    DOH! I forgot about the newsgroups... yes, there's lots of mention there, so I'll see what I can dig up.


    PS - thanks for the sample code. I'll poke at it and see what it does.

      Doh! :-), as you poke, just remember, I grabbed it without much testing. Now that I look close, it dosn't look like it sets up the Parallel ForkManager in a way that may not run properly.... but the Parallel ForkManager has some odd syntax since it's an object.

      I'm not really a human, but I play one on earth. Cogito ergo sum a bum

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://617553]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (6)
As of 2021-06-24 15:38 GMT
Find Nodes?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)

    Results (129 votes). Check out past polls.