Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Foreign directory search with LWP part 2?

by Anonymous Monk
on Apr 27, 2001 at 13:00 UTC ( #76024=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

so how does Yahoo or AltaVista do it?
  • Comment on Foreign directory search with LWP part 2?

Replies are listed 'Best First'.
Re: Foreign directory search with LWP part 2?
by larsen (Parson) on Apr 27, 2001 at 13:51 UTC
Re: Foreign directory search with LWP part 2?
by little (Curate) on Apr 27, 2001 at 13:25 UTC
    they use a porgram called spider that looks up for a file called "robots.txt" in the root dir of the specified (registered) domain. So they try to get "") Inside this file is specified what the spider may read and what directories he might search. But still not. Acces must be granted.
    You can read about this very good on the special section of these search engines, where is described how such a file must and can look like and how it works.
    CPAN has it

    Have a nice day
    All decision is left to your taste
Re: Foreign directory search with LWP part 2?
by suaveant (Parson) on Apr 27, 2001 at 16:37 UTC
    ... and what a spider does is that it looks at a specified page, and finds all the links on it. It then follows all of those links (at least the ones local to the site), and downloads them as well. you keep repeating this process until you have all the pages (min you have to keep track of which pages you have to prevent an infinite loop). But there are already module for that (though sometimes its nice to know what your modules are doing).
                    - Ant

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://76024]
Approved by root
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (3)
As of 2021-06-19 19:09 GMT
Find Nodes?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)

    Results (93 votes). Check out past polls.