Re: The Ethics of Webbots

[inman]

Your situation depends on two things. Firstly the relationship between the website owners and you. Secondly, what you do with the information.

If you are scraping the website of a competitor, someone who derives an income stream from the information or someone who has already paid someone else for the content on their website then you probably won't get very far. It is also the case that if you are re-selling the information that you get or not attributing the source of the information then you would be in trouble (breech of copyright, passing off etc.).

If, on the other hand, you are crawling a website that is otherwise in the public domain (e.g. government websites) then it may be worth getting in touch with the website owners and talking to them about it. Content owners are trying harder to provide machine to machine services such as web services, RSS feeds etc. A small licence fee later and you could end up with a web service feed rather than a web site scrape.

As a general note from a technical point of view - be nice when you are scraping/indexing.

• Index a site slowly - leaving a second or two between requests will take the pressure of the website (there's nothing like a DOS attack to annoy people). Varying the spacing in between calls also stops the requests clumping together in the website logs and drawing attention to your spider.
• Set the Agent header in your requests to include a contact e-mail so that the website admin can get in touch with you.
• Index a site only when necessary. The chances are that most of the site is staitic with only a small amount of info changing. The guiding principal is Less is more.

In terms of your relationship with your boss. You need to document your concerns and the approach that you are taking. Remember that if you can demonstrate that they knew what was happening then they get it in the neck and not you.