Re^3: Speeding up/parallelizing hundreds of HEAD requests

[perrin]

You mentioned caching the requests instead of actually doing them every time. This is a good way to do that. It's more efficient than a bunch of database queries, and makes it very easy to control timeouts on the cache, in case you only want to check every 2 hours or so.