From the text there, it sounds like more or less any link. However, when looking at the site without logging in, there are a great number of valid links.
For Perlmonks, it would probably be the Super Search, RAT (at least over a defined nesting depth), and nodelets with dynamic content (e.g. Other Users, CB) that would be "unlinked". But I think these are already blocked for robots.
So, probably not much to gain here :-/ especially given the fact that this concerns only "A HREF"s, not incoming links from elsewhere (or self-generated ones).
we don't want to completely prevent robots from spidering the site
Yes, that's understandable. Don't know how Fossil handles that. OTOH, they seem to be less well known, maybe as a result from exactly that 🙈
| [reply] |
