Re: Proxy link rotation

[OlegG]

Some time ago I writed LWP::UserAgent subclass, which main purpose is to automate proxy rotation for each request. It is not finished, not well tested and not documented yet, but you can try: LWP::UserAgent::Proxified
Simpliest variant of usage:

use LWP::UserAgent::Proxified;

my $ua = LWP::UserAgent::Proxified->new(
  proxylist => [
      ['http', 'https'] => 'http://10.0.0.1:1080',
      ['http', 'https'] => 'http://10.0.0.2:1080'
  ],
  proxyrand => 1, # choose random proxy for each request
# other lwp options goes here
);

[kazak]

I need to use 250-300 different proxies, all these proxies require authorization, also I need some special filter for disabling and enabling proxies in proxy-list (something like grey list),etc . Is there any mechanisms to implement these things within your class?

[OlegG]

I think you can

use strict;
use LWP::UserAgent::Proxified;

open( ROUTES, "<", "/etc/squid/repeater/lib/routes.cfg" )
    or die "open: $!";

my @proxylist;
while( <ROUTES> ) {
    s/#.*//;
    next if /^(\s)*$/;
    chomp;
    push @proxylist, http => "http://uname:passwd\@$_" if !/DISABLED/;
}
close(ROUTES);

my $ua = LWP::UserAgent::Proxified->new(
    agent => undef,
    proxylist => \@proxylist,
    proxyrand => 1
);

# do the job