Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Re: [OT] Ethical and Legal Screen Scraping

by tlm (Prior)
on Jul 26, 2005 at 03:16 UTC ( #478061=note: print w/replies, xml ) Need Help??

in reply to [OT] Ethical and Legal Screen Scraping

For example, suppose I write a little tool using LWP::UserAgent or WWW::Mechanize (rather than LWP::RobotUA or WWW::Mechanize::Polite ?, say) that simply collects a number of web pages for me while I sleep. Is it illegal or unethical for such a scraper to ignore robots.txt?

Whether it is legal or not I won't get into, but AFAIC, I don't have any ethical objection to such a tool as long as it doesn't impose a greater load on the target server(s) than you would if you were to perform the same task manually.

the lowliest monk

  • Comment on Re: [OT] Ethical and Legal Screen Scraping

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://478061]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (7)
As of 2020-03-30 20:24 GMT
Find Nodes?
    Voting Booth?
    To "Disagree to disagree" means to:

    Results (176 votes). Check out past polls.