Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re: [OT] Ethical and Legal Screen Scraping

by Anonymous Monk
on Jul 25, 2005 at 15:08 UTC ( #477863=note: print w/replies, xml ) Need Help??

in reply to [OT] Ethical and Legal Screen Scraping

If a commercial company sells a tool that allows non-programmer end users to write little screen scraping robots, is it unethical or illegal for such a product to not provide a mechanism to allow their end users to respect robots.txt?
First of all, I don't think the answer to the question depends on whether the tool is sold or not, or whether it's produced by a company that may, or may not, be commercial.

As for the legal status, like many other things on the internet, that will remain unclear for a long time. Jurisdiction concerning internet matters grows very slowly - the fact the internet doesn't care about our legal borders is a not unimportant factor in the slow growth of jurisdiction.

As for the ethical thing. If it doesn't allow a mechanism to respect robots.txt, it's certainly a bad product. And I guess selling bad products is unethical. It's certainly unethical for the end user to use such a tool though. (Well, IMO). It's the user of robot created that way that carries the responsibility - just like you carry the responsibility your car has headlights and brakes, the judge won't accept a defense of "but it was sold to me this way".

  • Comment on Re: [OT] Ethical and Legal Screen Scraping

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://477863]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (6)
As of 2020-04-09 11:20 GMT
Find Nodes?
    Voting Booth?
    The most amusing oxymoron is:

    Results (47 votes). Check out past polls.