Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

LWP::Parallel is slow downloader?

by dannoura (Pilgrim)
on Apr 13, 2005 at 12:57 UTC ( #447347=perlquestion: print w/ replies, xml ) Need Help??
dannoura has asked for the wisdom of the Perl Monks concerning the following question:

hi,

I have a script which downloads several hundreds of webpages. Not too long ago I switched from LWP to LWP::Parallel. The problem is that the downloading seems to go much slower than before. I tried use LWP::Debug qw(+); to try to find out what's causing the delays, but couldn't find anything useful. Are there any suggestions about where I could find something to help me figure out why the downloading is so slow?

p.s. I'm sending out 5 parallel requests each time.

Comment on LWP::Parallel is slow downloader?
Download Code
Replies are listed 'Best First'.
Re: LWP::Parallel is slow downloader?
by perrin (Chancellor) on Apr 13, 2005 at 13:31 UTC
    That's right, LWP::Parallel is slow. If you want to go fast, use forking or a different tool altogether. Things like HTTP::GHTTP or HTTP::MHTTP tend to be much faster than LWP.
Re: LWP::Parallel is slow downloader?
by webchalkboard (Scribe) on Apr 13, 2005 at 14:39 UTC

    Maybe you want to play around with the $pua->timeout (2); settings, or as the monk above said, fork off processes manually rather than using the LWP::Parallel, I found it a bit slow when I was using it ages ago as well.

    Learning without thought is labor lost; thought without learning is perilous. - Confucius
    WebChalkboard.com | For the love of art...
Re: LWP::Parallel is slow downloader?
by ghenry (Vicar) on Apr 13, 2005 at 13:32 UTC

    Could we see some code?

    Thanks.

    Walking the road to enlightenment... I found a penguin and a camel on the way.....
    Fancy a yourname@perl.me.uk? Just ask!!!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://447347]
Approved by tlm
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (13)
As of 2015-07-29 21:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (269 votes), past polls