Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re^3: Speeding up/parallelizing hundreds of HEAD requests

by perrin (Chancellor)
on Sep 16, 2007 at 20:06 UTC ( #639292=note: print w/replies, xml ) Need Help??


in reply to Re^2: Speeding up/parallelizing hundreds of HEAD requests
in thread Speeding up/parallelizing hundreds of HEAD requests

You mentioned caching the requests instead of actually doing them every time. This is a good way to do that. It's more efficient than a bunch of database queries, and makes it very easy to control timeouts on the cache, in case you only want to check every 2 hours or so.
  • Comment on Re^3: Speeding up/parallelizing hundreds of HEAD requests

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://639292]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (4)
As of 2021-01-28 06:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?