Anonymous Google Chrome browsers now under additional scrutiny

[Co-Rion]

If you are browing Perlmonks using Google Chrome and are not logged in, your browser needs to send the sec-ch-ua header, otherwise your anonymous access to the site is blocked.

We are constantly being hit by crawlers claiming to be Google Chrome, so this is a first step in the rat race.

[cavac]

I have the same problem on my own sites. I've added a blackhole that seems to, at least partially, mitigate the problem.

1. Added an entry in my robots.txt:
   

   User-agent: *
   Disallow: /secret/bla.php
   

2. Added a hidden link to the main page to an inbetween page that is never visibly linked anywhere. This mainly prevents that annoying pre-loading in chrome triggers anything.
3. The inbetween page links to /secret/bla.php
4. All IPs of clients navigating to /secret/bla.php get firewalled.

It's not perfect, but i was able to reduce bot traffic (no matter what UserAgent was set) by roughly 50%-80%. It's only a temporary win in the war against china and the silicon valley crowd, of course. But every little bit helps.

[LanX]

Well yes, a node behind the honeypot could automatically block the IP.

But probably it's better to redirect blacklisted IPs to some static and link-wise shallow fake content to feed them a little. ¹

Otherwise HTTP-Errors are easily detected, so someone might be triggered to improve the attack.

From my understanding they are using IP-farms to attack us and share the harvested links among them for the next requests. Like that we might trap far more IPs in our "honeyfarm"

¹) like a wget-mirror of the last week in a dedicated node-id range, there is a huge unused gap there, which we could reuse.

[cavac]

Hmm, for some (other) attacks on my server, my backend actually grabs the WHOIS information and tries to extract the network range, which then gets iptabled.

Of course, for my private server i can be a lot more aggresive in blocking large swathes of the internet.

[harangzsolt33]

Wow, that's really clever! :)

[cavac]

Wasn't my idea. Found the basics on some forum (can't remember which) and then just iterated over the idea. There are some other "blackhole" URIs as well, mostly stuff check by bots searching for vulnerabilities. You know, Wordpress config pages and such...

[LanX]

What's happening to me when using Chrome - be it mobile or desktop - is that I get a 403 when using another domain, i.e. not being logged in AND I'm having JS disabled by default. ¹

prohibited
You don't have permission to access this resource.

Now the problem I see:

What about new users coming here with Chrome and no JS for the first time. Are they able to create a new account if all pages are prohibited?

Or am I missing something?

¹) not sure how this is related to the sec-ch-ua header.

update

so when checking headers in the console it looks like sec-ch-ua only appears when JS is on:

sec-ch-ua:   "Not A(Brand";v="8", "Chromium";v="132"

so AnoMonks just need to make sure JS is activated or create a login when they want JS to be off.

[Corion]

Yes. Such is life.

[Co-Rion]

If you're sending a Chrome User-Agent, you now also need to make sure that your Referer header and the Sec-Fetch-Site header agree.