Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re^3: What Tools Do You Use With WWW::Mechanize

by Limbic~Region (Chancellor)
on Oct 04, 2011 at 18:43 UTC ( #929637=note: print w/ replies, xml ) Need Help??


in reply to Re^2: What Tools Do You Use With WWW::Mechanize
in thread What Tools Do You Use With WWW::Mechanize

OfficeLinebacker,
Have you seen WWW::Mechanize::Sleepy? Personally, I use something along the lines of:

# Sleep a random interval between $duration and 2 * $duration - 1 unit sub rest { my ($duration) = @_; sleep $duration; sleep rand($duration); } sub fetch_page { my ($mech, $action, $target, $max, $duration) = @_; for (1 .. $max) { rest($duration); eval {$mech->$action($target);}; return if ! $@ && $mech->status == OK; } die "Failed to fetch '$url' after '$max' attempts\n"; }

Of course, if you want to allow for HTTP redirects then you will need to change status == OK to include acceptable HTTP codes. Additionally, if you use Time::HiRes to overload sleep, you can easily sleep for partial minutes. In truth, I typically use milliseconds.

Cheers - L~R


Comment on Re^3: What Tools Do You Use With WWW::Mechanize
Download Code
Re^4: What Tools Do You Use With WWW::Mechanize
by OfficeLinebacker (Chaplain) on Oct 05, 2011 at 13:34 UTC
    I hadn't even heard of WWW::Mechanize::Sleepy! That's great. During debugging I get impatient but once the product is rolled out it only has to run once a day or so, so using seconds is fine. Also I think you mean "you can easily sleep for partial seconds," right?

    I like computer programming because it's like Legos for the mind.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://929637]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (6)
As of 2015-07-05 04:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (60 votes), past polls