Re^10: Unable to connect

[jdporter]

Thanks. I didn't know about that. So monitoring the user-agent strings, some interesting things pop out:

1. Go-http-client/1.1
2. Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
3. Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [e-mail])
4. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)
5. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36
6. Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [e-mail])
7. Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; [e-mail])
8. Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.6998.165 Mobile Safari/537.36 (compatible; GoogleOther)
9. Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.6998.165 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Some clearly say "bot" (or, in a couple cases, "spider"), which is nice. What I'm a little concerned about is #8, "GoogleOther". What does that mean? It's also coming from 66.249.*, just like the "Googlebot" ones. We don't want to keep Google from indexing the site, we just don't want it to hit Super Search (too often).

A bunch of different ones occur in the log (which, btw, goes back to Sept. 2021)...

AhrefsBot
Amazonbot
AntBot
Applebot
BLEXBot
Bytespider
CCBot
ClaudeBot
DataForSeoBot
DotBot
GPTBot
GoogleOther
Googlebot
PetalBot
SPIDER
SemrushBot
SEOkicks
SpiderLing
TikTokSpider
YandexBot
bingbot

[hippo]

What I'm a little concerned about is #8, "GoogleOther". What does that mean?

It's a google bot but not search-related. See https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers#googleother.

Of your list of short useragents I would (and do) block all bar Googlebot, YandexBot, CCBot, SpiderLing and bingbot. Many of them are IME incredibly badly behaved.