Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Re: Checking links with LWP::UserAgent

by lhoward (Vicar)
on Feb 26, 2002 at 13:24 UTC ( #147550=note: print w/replies, xml ) Need Help??

in reply to Checking links with LWP::UserAgent

There's nothing wrong with your code per-se. Its just that it is checking each site sequentially; which requires a DNS lookup for each site, connect to each site, etc... What you need to do to improve performance is to be able to check multiple sites in parallel. Fortunately, there is a module that extends LWP for just this type of situation: LWP::Parallel.

Replies are listed 'Best First'.
Re: Re: Checking links with LWP::UserAgent
by wilstephens (Acolyte) on Feb 26, 2002 at 13:51 UTC
    Thanks! I'll look into this!

    Wiliam Stephens <>
Re: Re: Checking links with LWP::UserAgent
by webfiend (Vicar) on Feb 26, 2002 at 23:49 UTC

    I was going to offer the same advice, but it looks like you beat me to it. LWP::Parallel is just the ticket for getting a lot of links in short order.

    "All you need is ignorance and confidence; then success is sure." -- Mark Twain
Re: Re: Checking links with LWP::UserAgent
by wilstephens (Acolyte) on Mar 08, 2002 at 10:37 UTC
    OK. Does anyone know how I could go about converting the above code to use LWP::Parallel then? I'm afraid that I will run it and it will fire off and look through the 300+ sites all in parallel at the same time? Is there a way to limit it? -- Wiliam Stephens <>
      The docs have some good examples. You set the option:
      to set how many requests it will handle in parallel.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://147550]
[thezip]: Our cat is single-handedly (pawed-ly?) solving our field mouse problem. About four critters in the last week alone.
[thezip]: Amazing speed and stealth in that one.
[GotToBTru]: You can't tunafish, but some seafood companies can tunafish. More fun with English.

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (15)
As of 2017-11-17 19:48 GMT
Find Nodes?
    Voting Booth?
    In order to be able to say "I know Perl", you must have:

    Results (272 votes). Check out past polls.