Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer

Re^3: [OT] Ethical and Legal Screen Scraping

by wazoox (Prior)
on Jul 26, 2005 at 09:46 UTC ( #478115=note: print w/replies, xml ) Need Help??

in reply to Re^2: [OT] Ethical and Legal Screen Scraping
in thread [OT] Ethical and Legal Screen Scraping

You seem to be confused about ethics. Behaving ethically is not defined by anyone else's ability to prove that you did or did not behave ethically.

Looks like you didn't understand me. You're confusing the "ethic" aspect of my answer with the "legal" aspect. I mean "enforcing legally the respect of robots.txt is abusive and unethical.". I mean too "I can't imagine that you'd be legally in trouble for web-spidering, for personal use, without taking care of robots.txt, except in China or Iran.". I never meant "Go suck all websites you want to and don't bother", ever.

I think too the generalisation "if you use a web-spidering program without taking notice of robots.txt, you're ethically wrong", is questionable. Ethics aren't that simple. Perhaps he's a political refugee trying to extract important information from some hostile hidden website. Is he "right" or "wrong"?

  • Comment on Re^3: [OT] Ethical and Legal Screen Scraping

Replies are listed 'Best First'.
Re^4: [OT] Ethical and Legal Screen Scraping
by tilly (Archbishop) on Jul 26, 2005 at 22:44 UTC
    I had misunderstood that you were saying that the law itself is unethical. I think that distinction was probably lost on many others as well.

    However, as I think the subsequent discussion clarified, my views are not as simple as saying that thou shalt always do what robots.txt says. (Paying attention to it does not mean that you necessarily agree or do what it says...)

      Well, I probably wasn't clear enough myself. And fortunately it's not yet exactly a law but a jurisprudence AFAIK, which is somewhat better (if a law comes someday to draw the line...)

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://478115]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (5)
As of 2020-04-05 01:56 GMT
Find Nodes?
    Voting Booth?
    The most amusing oxymoron is:

    Results (33 votes). Check out past polls.