Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Re: [OT] Ethical and Legal Screen Scraping

by brian_d_foy (Abbot)
on Jul 25, 2005 at 17:06 UTC ( #477901=note: print w/replies, xml ) Need Help??

in reply to [OT] Ethical and Legal Screen Scraping

The robots.txt file can specify sections that apply to a particular user-agent or to all user-agents. It can be for any user-agent, including one that you write yourself.

You know about robots.txt, and you know that it is a statement of the website's operator about how they want an automated agent to access their website. You have to decide for yourself whether ignoring those instructions violates your ethics. However, I tend to think that if you have to ask the question, you already know there is an ethical problem.

For the legal questions, you'll have to talk to a lawyer who can handle the various local (or international) laws that may apply. However, I'd much rather you obey the spirit of the mechanism rather than the letter of the law. If things get too out of hand, we'll just get more regulation.

brian d foy <>
  • Comment on Re: [OT] Ethical and Legal Screen Scraping

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://477901]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (4)
As of 2020-04-04 03:38 GMT
Find Nodes?
    Voting Booth?
    The most amusing oxymoron is:

    Results (32 votes). Check out past polls.