in reply to "Web Automation" -- your input is greatly desired!
Under Spiders, don't forget those that use LWP and such to read newsfeeds or straight HTML to keep themselves informed. jcwren has a lot of tools for checking your XP here on PerlMonks.
This might be more of a research topic, however, I've been finding that a lot of websites might be faking the referrer as some sort of secret ad to the webmaster. I keep getting one or two hits for different sites, but I never find anything that links to my site.
- Just a thought:
- Spiders
- News gathers
- Broken link finders
- Bad Spiders that Overindex/Look for Holes
- Automation
- Testing CGI scripts
- Checking for updated content
- others...
- Scraping
- Using Perl and the HTML:: Modules
- Making Clean HTML (easy to parse/scrape)
- Using data-oriented methods (XML/RSS)
This list is nowhere near complete. If I'm off on something, post a reply and set me straight.
John J Reiser
newrisedesigns.com
In Section
Seekers of Perl Wisdom