Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked

Re: Speeding up/parallelizing hundreds of HEAD requests

by perrin (Chancellor)
on Sep 16, 2007 at 18:15 UTC ( #639283=note: print w/replies, xml ) Need Help??

in reply to Speeding up/parallelizing hundreds of HEAD requests

The easiest way to do the caching is to use Cache::FastMmap. It's very fast and you can set the timeout to be whatever you like.

Caching isn't very useful if the requests vary widely, i.e. if people don't tend to request the same page again within your timeout period. In that case, you really would want to run these requests in parallel. Parallel::ForkManager probably is the easiest way to do this, but forking a hundred processes may hurt a bit. The non-blocking I/O approach is more complicated but easier on your system. That's what LWP::Parallel does. There are other implementations, like HTTP::Async. All of them are more complicated than vanilla LWP, and I think that's unavoidable.

  • Comment on Re: Speeding up/parallelizing hundreds of HEAD requests

Replies are listed 'Best First'.
Re^2: Speeding up/parallelizing hundreds of HEAD requests
by hacker (Priest) on Sep 16, 2007 at 19:58 UTC

    I must have missed something in your reply.. how exactly is a local shared memory cache on a mmap'd file, going to help me speed up external HEAD requests to dozens/hundreds of separate resources over HTTP?

    For access to my templates and local files, sure, I can see how this would help (but so does Memoize, HTML::Template::Compiled, and so on), but I'm not sure where this helps speed up remote requests.

      You mentioned caching the requests instead of actually doing them every time. This is a good way to do that. It's more efficient than a bunch of database queries, and makes it very easy to control timeouts on the cache, in case you only want to check every 2 hours or so.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://639283]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (5)
As of 2021-01-24 10:24 GMT
Find Nodes?
    Voting Booth?