Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re^2: use LWP::Simple slows script down.

by Jesse Smith (Novice)
on Feb 11, 2014 at 18:14 UTC ( #1074462=note: print w/ replies, xml ) Need Help??


in reply to Re: use LWP::Simple slows script down.
in thread use LWP::Simple slows script down.

Is it normal for it to be very slow when it's only getting a text file with a number, and a text file with an IP address, with the files on the same domain?


Comment on Re^2: use LWP::Simple slows script down.
Re^3: use LWP::Simple slows script down.
by davido (Archbishop) on Feb 11, 2014 at 19:07 UTC

    The speed with which a remote server responds to an HTTP request is not within the scope of your local Perl program. The only way that your local program could have an effect on how the server responds would be to know if the remote server uses some metric to selectively throttle requests, and if so, to do your best to program around that remote-servers behavior.

    In other words, in all probability, the remote server is just slow sometimes. However, it's possible that the remote server is detecting your UserAgent by name (or by IP, or by lack of accepting cookies, or some other metric), and selectively choosing to slow down your requests. Either way, it's not an LWP::Simple issue, or even a Perl issue, the remote server is the one who decides how long requests take, not LWP::Simple. If it does turn out that you're being throttled based on some metric, the solution would be to uncover what criteria are resulting in the server slowing down its responses to your requests, and then try to not fall within those criteria. Maybe you need to accept cookies. Maybe you need to rename your user agent. Maybe you need to originate from different IP's, or to slow down the rate at which you're making requests.

    Your best bet, assuming you're living within the TOS of the server you are hitting, is to ask the administrator of that server why his server is slow, and if it's because you're falling into some criteria for the engagement of a safety mechanism, work with the admin to find ways to avoid tripping the behavior.


    Dave

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1074462]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (7)
As of 2014-09-19 04:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (129 votes), past polls