Re^7: Perlmonks site has become far too slow (AnoServer/Load Balancer)

[LanX]

> Since we are working with multiple servers I'm pessimistic about using the filesystem for caching, I think it's far easier to implement in the database server.

Unless it's possible to direct all AnoMonk requests to one server only. This would have the huge advantage that other web-servers wouldn't be impacted by increased bot activities.

OTOH update events could happen on other servers and would need to be logged via the DB, so that the "Ano-Server" could discard the cached page.

update

Or storing the cache in a shared file system, unfortunately I have no experience to tell how efficient this is.

But in this case the Ano-Server wouldn't need to run Everything, just a web-server (apache/Nginx/...) distributing static files.

The updates of the static files are comparatively rare and could be easily handled by the dynamic servers.

Alas some pages can't be static like log-in, chatterbox, other users. So the load-balancer or a redirection rule has to bring Ano-monks to the real servers.

Anyway ... no matter if it's a static or dynamic server, setting up a dedicated server for Anomonks would improve many issues instantly.

[LanX]

Regarding the "macro" caching of anonymously viewed pages.

After some searching I think the most robust approach is to set up one or more reverse proxies to handle the bots and SEO.

They can cache all static requests locally and rules can check the cookies and relay logged in users to the upstream servers for dynamic content.

Cache management is also included, nginx for instance can receive "purge" requests of cached urls remotely via web requests, which means those proxies don't even need to be on the same box.

And this doesn't even need to be triggered directly from inside the Everything engine.

Running an external cronjob checking for new nodes via XML ticker (like every x minutes) could be implemented immediately and update all proxies. ¹

So no need to patch anything inside the monastery except server settings.

I think "Micro caching" to speed up the engine's most frequent evals and queries is also possible but would certainly require patching the Everything codebase. But this would require good benchmarking and is less relevant for our "bot war".

¹) of course querying the DB directly for new and updated nodes and votes would be even better.

My point is to show how flexible and available this strategy would be while immediately shielding off our engines from DNS attacks.