in reply to Re^2: use LWP::Simple slows script down.
in thread use LWP::Simple slows script down.
The speed with which a remote server responds to an HTTP request is not within the scope of your local Perl program. The only way that your local program could have an effect on how the server responds would be to know if the remote server uses some metric to selectively throttle requests, and if so, to do your best to program around that remote-servers behavior.
In other words, in all probability, the remote server is just slow sometimes. However, it's possible that the remote server is detecting your UserAgent by name (or by IP, or by lack of accepting cookies, or some other metric), and selectively choosing to slow down your requests. Either way, it's not an LWP::Simple issue, or even a Perl issue, the remote server is the one who decides how long requests take, not LWP::Simple. If it does turn out that you're being throttled based on some metric, the solution would be to uncover what criteria are resulting in the server slowing down its responses to your requests, and then try to not fall within those criteria. Maybe you need to accept cookies. Maybe you need to rename your user agent. Maybe you need to originate from different IP's, or to slow down the rate at which you're making requests.
Your best bet, assuming you're living within the TOS of the server you are hitting, is to ask the administrator of that server why his server is slow, and if it's because you're falling into some criteria for the engagement of a safety mechanism, work with the admin to find ways to avoid tripping the behavior.
Dave