Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re^2: (OT) Robots disallow

by jhourcle (Prior)
on Apr 01, 2009 at 18:23 UTC ( #754770=note: print w/ replies, xml ) Need Help??


in reply to Re: (OT) Robots disallow
in thread (OT) Robots disallow

If there's a specific signature of the bot that you're having problems with (a specific domain, or an identifiable user-agent), look into configuring your webserver to send them alternative content.

Back when it was easy to identify e-mail harvesters, I'd send them to a CGI that slowly feeds them bogus e-mail addresses and the abuse address from the netblock they're coming from. These days, most harvesters are coming from botnets, so the abuse one isn't so useful. (and yes, it _is_ a Perl script)


Comment on Re^2: (OT) Robots disallow

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://754770]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (11)
As of 2015-07-02 19:01 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (44 votes), past polls