Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re: [OT] Ethical and Legal Screen Scraping

by brian_d_foy (Abbot)
on Jul 25, 2005 at 17:06 UTC ( #477901=note: print w/replies, xml ) Need Help??


in reply to [OT] Ethical and Legal Screen Scraping

The robots.txt file can specify sections that apply to a particular user-agent or to all user-agents. It can be for any user-agent, including one that you write yourself.

You know about robots.txt, and you know that it is a statement of the website's operator about how they want an automated agent to access their website. You have to decide for yourself whether ignoring those instructions violates your ethics. However, I tend to think that if you have to ask the question, you already know there is an ethical problem.

For the legal questions, you'll have to talk to a lawyer who can handle the various local (or international) laws that may apply. However, I'd much rather you obey the spirit of the mechanism rather than the letter of the law. If things get too out of hand, we'll just get more regulation.

--
brian d foy <brian@stonehenge.com>
  • Comment on Re: [OT] Ethical and Legal Screen Scraping

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://477901]
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (3)
As of 2018-01-19 04:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How did you see in the new year?










    Results (215 votes). Check out past polls.

    Notices?