Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re: "Web Automation" -- your input is greatly desired!

by newrisedesigns (Curate)
on May 05, 2003 at 20:28 UTC ( #255746=note: print w/ replies, xml ) Need Help??


in reply to "Web Automation" -- your input is greatly desired!

Under Spiders, don't forget those that use LWP and such to read newsfeeds or straight HTML to keep themselves informed. jcwren has a lot of tools for checking your XP here on PerlMonks.

This might be more of a research topic, however, I've been finding that a lot of websites might be faking the referrer as some sort of secret ad to the webmaster. I keep getting one or two hits for different sites, but I never find anything that links to my site.

    Just a thought:
  • Spiders
    • News gathers
    • Broken link finders
    • Bad Spiders that Overindex/Look for Holes
  • Automation
    • Testing CGI scripts
    • Checking for updated content
    • others...
  • Scraping
    • Using Perl and the HTML:: Modules
    • Making Clean HTML (easy to parse/scrape)
    • Using data-oriented methods (XML/RSS)

This list is nowhere near complete. If I'm off on something, post a reply and set me straight.

John J Reiser
newrisedesigns.com


Comment on Re: "Web Automation" -- your input is greatly desired!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://255746]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (9)
As of 2015-07-08 08:34 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (98 votes), past polls