Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

LWP::Parallel is slow downloader?

by dannoura (Pilgrim)
on Apr 13, 2005 at 12:57 UTC ( #447347=perlquestion: print w/ replies, xml ) Need Help??
dannoura has asked for the wisdom of the Perl Monks concerning the following question:


I have a script which downloads several hundreds of webpages. Not too long ago I switched from LWP to LWP::Parallel. The problem is that the downloading seems to go much slower than before. I tried use LWP::Debug qw(+); to try to find out what's causing the delays, but couldn't find anything useful. Are there any suggestions about where I could find something to help me figure out why the downloading is so slow?

p.s. I'm sending out 5 parallel requests each time.

Comment on LWP::Parallel is slow downloader?
Download Code
Replies are listed 'Best First'.
Re: LWP::Parallel is slow downloader?
by perrin (Chancellor) on Apr 13, 2005 at 13:31 UTC
    That's right, LWP::Parallel is slow. If you want to go fast, use forking or a different tool altogether. Things like HTTP::GHTTP or HTTP::MHTTP tend to be much faster than LWP.
Re: LWP::Parallel is slow downloader?
by webchalkboard (Scribe) on Apr 13, 2005 at 14:39 UTC

    Maybe you want to play around with the $pua->timeout (2); settings, or as the monk above said, fork off processes manually rather than using the LWP::Parallel, I found it a bit slow when I was using it ages ago as well.

    Learning without thought is labor lost; thought without learning is perilous. - Confucius | For the love of art...
Re: LWP::Parallel is slow downloader?
by ghenry (Vicar) on Apr 13, 2005 at 13:32 UTC

    Could we see some code?


    Walking the road to enlightenment... I found a penguin and a camel on the way.....
    Fancy a Just ask!!!

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://447347]
Approved by tlm
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (8)
As of 2015-11-26 15:14 GMT
Find Nodes?
    Voting Booth?

    What would be the most significant thing to happen if a rope (or wire) tied the Earth and the Moon together?

    Results (701 votes), past polls