Re^6: Super search use DuckDuckGo link broken

[ikegami]

IIRC, something was added to specify that perlmonks.org or www.perlmonks.org is the official domain, so no surprise you get better result from that one.

[LanX]

Yes, see the corresponding robots.txts

Like

• https://perlmonks.com/robots.txt

  # Please only spider https://www.perlmonks.org not https://perlmonks.c
  +om
  User-agent: *
  Disallow: /
  

• https://www.perlmonks.org/robots.txt

  # Be kind. Wait between fetches longer than each fetch takes.
  User-agent: *
  Disallow: /bare/
  Disallow: /mobile/
  Crawl-Delay: 20
  

[LanX]

According to the robots.txt specifications I found at Google, it's possible to exclude "orthogonal" pages like &displaytype=print or ;displaytype=edithistory with wildcards.

Any reason not to add

• Disallow: /*displaytype=

to the list? (Untested)

Bing also suggests adding noindex-meta tags to the pages.

On a tangent

Ideally robots would be presented with a page without nodelets, but I'm not aware of an efficient solution, except checking the user-agent before building the page.

[Corion]

These links are already tagged with rel="nofollow", so Google shouldn't be crawling these, nor should any other bots. Except a lot of them do, so I'm not sure if spending any effort on divining bot behaviours is time well spent.