Re^3: What Tools Do You Use With WWW::Mechanize

[Limbic~Region]

in reply to Re^2: What Tools Do You Use With WWW::Mechanize
in thread What Tools Do You Use With WWW::Mechanize

OfficeLinebacker,
Have you seen WWW::Mechanize::Sleepy? Personally, I use something along the lines of:

# Sleep a random interval between $duration and 2 * $duration - 1 unit
sub rest {
    my ($duration) = @_;
    sleep $duration;
    sleep rand($duration);
}

sub fetch_page {
    my ($mech, $action, $target, $max, $duration) = @_;
    for (1 .. $max) {
        rest($duration);
        eval {$mech->$action($target);};
        return if ! $@ && $mech->status == OK;
    }
    die "Failed to fetch '$url' after '$max' attempts\n";
}

Of course, if you want to allow for HTTP redirects then you will need to change status == OK to include acceptable HTTP codes. Additionally, if you use Time::HiRes to overload sleep, you can easily sleep for partial minutes. In truth, I typically use milliseconds.

[OfficeLinebacker]

I hadn't even heard of WWW::Mechanize::Sleepy! That's great. During debugging I get impatient but once the product is rolled out it only has to run once a day or so, so using seconds is fine. Also I think you mean "you can easily sleep for partial seconds," right?