Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

LWP::Parallel is slow downloader?

by dannoura (Pilgrim)
on Apr 13, 2005 at 12:57 UTC ( #447347=perlquestion: print w/ replies, xml ) Need Help??
dannoura has asked for the wisdom of the Perl Monks concerning the following question:

hi,

I have a script which downloads several hundreds of webpages. Not too long ago I switched from LWP to LWP::Parallel. The problem is that the downloading seems to go much slower than before. I tried use LWP::Debug qw(+); to try to find out what's causing the delays, but couldn't find anything useful. Are there any suggestions about where I could find something to help me figure out why the downloading is so slow?

p.s. I'm sending out 5 parallel requests each time.

Comment on LWP::Parallel is slow downloader?
Download Code
Re: LWP::Parallel is slow downloader?
by perrin (Chancellor) on Apr 13, 2005 at 13:31 UTC
    That's right, LWP::Parallel is slow. If you want to go fast, use forking or a different tool altogether. Things like HTTP::GHTTP or HTTP::MHTTP tend to be much faster than LWP.
Re: LWP::Parallel is slow downloader?
by ghenry (Vicar) on Apr 13, 2005 at 13:32 UTC

    Could we see some code?

    Thanks.

    Walking the road to enlightenment... I found a penguin and a camel on the way.....
    Fancy a yourname@perl.me.uk? Just ask!!!
Re: LWP::Parallel is slow downloader?
by webchalkboard (Scribe) on Apr 13, 2005 at 14:39 UTC

    Maybe you want to play around with the $pua->timeout (2); settings, or as the monk above said, fork off processes manually rather than using the LWP::Parallel, I found it a bit slow when I was using it ages ago as well.

    Learning without thought is labor lost; thought without learning is perilous. - Confucius
    WebChalkboard.com | For the love of art...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://447347]
Approved by tlm
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (4)
As of 2014-12-28 15:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (182 votes), past polls