Re^2: What Tools Do You Use With WWW::Mechanize

[OfficeLinebacker]

in reply to Re: What Tools Do You Use With WWW::Mechanize
in thread What Tools Do You Use With WWW::Mechanize

Greetings, esteemed monks!

To reply to my own concern, I came up with this for generating wait times between link clicking/back() calls in a Mech script. What do you think?

#!/usr/bin/perl --
use strict;
use warnings;

my $i1 = int(rand(5)+1);
my $i2 = int(rand(2));
my $i = 0;

while ($i<10){
    print "$i1: $i2\n";
    my $interval = $i1 + ($i2*$i1);
    print "waiting for $interval seconds...\n";
    sleep($interval);
    $i1 = int(rand(5)+1);
    $i2 = int(rand(2));
    $i++;
}

Sample output:

1: 0
waiting for 1 seconds...
4: 0
waiting for 4 seconds...
1: 1
waiting for 2 seconds...
4: 1
waiting for 8 seconds...
4: 0
waiting for 4 seconds...
4: 1
waiting for 8 seconds...
4: 0
waiting for 4 seconds...
5: 0
waiting for 5 seconds...
1: 1
waiting for 2 seconds...
1: 1
waiting for 2 seconds...

[Limbic~Region]

OfficeLinebacker,
Have you seen WWW::Mechanize::Sleepy? Personally, I use something along the lines of:

# Sleep a random interval between $duration and 2 * $duration - 1 unit
sub rest {
    my ($duration) = @_;
    sleep $duration;
    sleep rand($duration);
}

sub fetch_page {
    my ($mech, $action, $target, $max, $duration) = @_;
    for (1 .. $max) {
        rest($duration);
        eval {$mech->$action($target);};
        return if ! $@ && $mech->status == OK;
    }
    die "Failed to fetch '$url' after '$max' attempts\n";
}

Of course, if you want to allow for HTTP redirects then you will need to change status == OK to include acceptable HTTP codes. Additionally, if you use Time::HiRes to overload sleep, you can easily sleep for partial minutes. In truth, I typically use milliseconds.

[OfficeLinebacker]

I hadn't even heard of WWW::Mechanize::Sleepy! That's great. During debugging I get impatient but once the product is rolled out it only has to run once a day or so, so using seconds is fine. Also I think you mean "you can easily sleep for partial seconds," right?