robots.txt for www.*.org broken

by Anonymous Monk

on Mar 27, 2026 at 12:29 UTC ( [id://11167483]=monkdiscuss: print w/replies, xml )

Need Help??

cat robots.txt

# Be kind. Wait between fetches longer than each fetch takes.
...
Disallow: /*node_id=3989nCrawl-Delay: 20
[download]

please note the 'n' instead of '\n' in the last line

Cheers
LanX

(via Tor, IP probably blocked because of greedy downloads ;-)

Comment on robots.txt for www.*.org broken Download Code

Domain Nodelet^?

Node Status^?

node history
Node Type: monkdiscuss [id://11167483]
Approved by LanX
help

Chatterbox^?

How do I use this? • Last hour • Other CB clients

Other Users^?

Others drinking their drinks and smoking their pipes about the Monastery: (4)

As of 2026-04-16 02:26 GMT

Sections^?

Information^?

Find Nodes^?

Leftovers^?

Voting Booth^?

No recent polls found

Notices^?

	• hippo	‥ epoptai's answer Re: how do I set a cookie and redirect was blessed by hippo!
	• erzuuli	‥ Anonymous Monks are no longer allowed to use Super Search, due to an excessive use of this resource by robots.