Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer

Web Spidering module?

by elbie (Deacon)
on Sep 23, 2001 at 02:05 UTC ( #114103=perlquestion: print w/ replies, xml ) Need Help??
elbie has asked for the wisdom of the Perl Monks concerning the following question:

Is there an existing module for spidering a website and returning a list of URLs? I found this but it would be nice if I could just have a module that does all the dirty work for me so I don't have to worry about it.


Replies are listed 'Best First'.
Re: Web Spidering module?
by charnos (Friar) on Sep 23, 2001 at 02:26 UTC
    While I've never used the module myself, WWW::Robot seems to be exactly what you are looking for. It seems to be a decent solution, but it has alot of external module dependencies. Good luck spidering!
Re: Web Spidering module?
by Starky (Chaplain) on Sep 23, 2001 at 10:46 UTC
    I can't seem to dig up the specific reference, but there was a Perl Journal article awhile back which, if I recall correctly, had a 4 line web spider.

    If someone out there could dig that up, it would be worth a read if only for amusement value :-)

        I found HTML::LinkExtor (also used in the one line spider mentioned above) on the weekend, and it was quite helpful.

        Thanks everybody!


Re: Web Spidering module?
by jryan (Vicar) on Sep 23, 2001 at 02:22 UTC
    Believe it or not, I started working on one just the other day... I'll post it when I am done (which will probably be the next chance I get a large chunk of time to code). However, if anyone knows if something like this is already in existence, please let me know so I can work on something else :)

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://114103]
Approved by root
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (3)
As of 2016-08-27 10:27 GMT
Find Nodes?
    Voting Booth?
    The best thing I ever won in a lottery was:

    Results (379 votes). Check out past polls.