Re^4: Can't call method "proxy" on an undefined value at

[Anonymous Monk]

Thanks. I see my error. I've modified the code and I can pull the proxy from the list but I'm getting the following error, "Proxy must be specified as absolute URI; '1.0.251.108:8080' is not at" with the code below:

open(my $fh, '<', 'cities.txt') or die $!;
    print $fh;
    open(my $prx, '<', 'proxies.txt') or die $!;
    print $prx;
    
    mkdir 'Bing', 0755;
    mkdir 'Bing/1Parsed/', 0755;
    mkdir 'Bing/1Parsed/Html/', 0755;
    
    chomp(my @cities = <$fh>);
    close($fh);
    
    chomp(my @prx = <$prx>);
    print $prx;
    close($prx);

    open($fh, '<', 'keywords.txt') or die $!;

    for my $city (@cities) {
        seek($fh, 0, 0);

        while (my $keywords = <$fh>) {
            chomp($keywords);
            print "$city $keywords\n";

    my $xml1 = $link1 . $city ."+". $keywords . $link2 . $city ."+". $
+keywords . $link3;
    #my $xmla = $link3 . $row . ".com";
    #my $xmlx = $link4 . $row;

    my $filename1 = "Bing/".($city)."_". ($keywords) . ".html";
    open my $fh1, ">", $filename1 or die("Could not open file. $!");
    
    #toggle proxy selection
    
    my $ua = LWP::UserAgent->new; 
    $ua->agent('Mozilla/8.0');

    for my $p (@prx){
        print "Current proxy:$p\n";
        $ua->proxy('http', $p);
         # Use this UA/Proxy to fetch something....
    }

I've read the can LPW info but I can't see how to fix this.

[NetWallah]

Two mistakes:

* You are missing the square brackets around the [http]

* You do not have a proper URL in $p

Try this:

  $ua->proxy(['http'], "http://$p");

[Anonymous Monk]

All of the help you have given is greatly appreciated. I'm just a newbie trying to do the best I can. One last question: So I'm trying to put it all together and read three different files and query through the proxy with the values pulled. Basically something like this: 1)use proxy one <value1 for file> 2)to search URL containing keyword<value 1 from 2nd file> + city<value 3 from 3rd file>

I have the for and the while that grabs the keyword and city from their respective files and it work perfectly but I can't seem to figure out how to put the proxy piece in. Nesting a for in a for doesn't seem to work and with the current code I'm not getting all three values changing as expected. Just the proxy value. Here's what I have

open(my $fh, '<', 'cities.txt') or die $!;
    print $fh;
    open(my $prx, '<', 'proxies.txt') or die $!;
    print $prx;
    
    mkdir 'Bing', 0755;
    mkdir 'Bing/1Parsed/', 0755;
    mkdir 'Bing/1Parsed/Html/', 0755;
    
    chomp(my @cities = <$fh>);
    close($fh);
    
    chomp(my @prx = <$prx>);
    print $prx;
    close($prx);

    open($fh, '<', 'keywords.txt') or die $!;

    for my $city (@cities) {
        seek($fh, 0, 0);

        while (my $keywords = <$fh>) {
            chomp($keywords);
            print "$city $keywords\n";
            
            #toggle proxy selection
    
            my $ua = LWP::UserAgent->new; 
            $ua->agent('Mozilla/8.0');

            for my $p (@prx){
                print "Current proxy:$p\n";
                $ua->proxy(['http'], "http://$p");
                 # Use this UA/Proxy to fetch something....
                 my $xml1 = $link1 . $city ."+". $keywords . $link2 . 
+$city ."+". $keywords . $link3;
                 #my $xmla = $link3 . $row . ".com";
                 #my $xmlx = $link4 . $row;

                 my $filename1 = "Bing/".($city)."_". ($keywords) . ".
+html";
                 open my $fh1, ">", $filename1 or die("Could not open 
+file. $!");
                 
                 
                my $xml2 = get $xml1;
                 print $xml1;
                 print "\n";
                 print $fh1 $xml2;

    
    
    
    }
}
}

When it's late I'm often not as clear headed ass I'd like to be as evidenced by my posts tonight!

[poj]

You might find it cleared using the same approach for all your loops and print statement to check what is happening. Note I have added a number to the output filename to create a different file for each proxy.

#!perl
use strict;
use LWP::UserAgent;
use File::Path 'make_path';

my $link1='';
my $link2='';
my $link3='';

# make directories
my $dir = 'Bing/1Parsed/Html/';
unless (-d $dir){
  make_path($dir, { verbose=>1, mode=>0755 });
}

# keywords
open my $fh, '<', 'keywords.txt' or die $!;
chomp(my @keywords = <$fh>);
close $fh;

# cities
open my $fh, '<', 'cities.txt' or die $!;
chomp(my @cities = <$fh>);
close $fh;

# proxies
open my $fh, '<', 'proxies.txt' or die $!;
chomp(my @proxies = <$fh>);
close $fh;
  
# create useragent
my $ua = LWP::UserAgent->new; 
$ua->agent('Mozilla/8.0');

# get pages for each city, keyword, proxy 
for my $city (@cities) {
  for my $keyword (@keywords){
    print "City :    [$city]\nKeyword: [$keyword]\n";

    for my $i (0..$#proxies){
      print "Current proxy:$proxies[$i]\n";

      # Use this UA/Proxy to fetch something....
      $ua->proxy(['http'], 'http://'.$proxies[$i]);     
      my $url = join '',$link1 . $city ."+". $keyword,
                        $link2 . $city ."+". $keyword,
                        $link3;
      my $response = $ua->get($url);
      print "getting $url\n";      
      if ($response->is_success) {
        my $filename = "Bing/${city}_${keyword}_${i}.html";
        print "Creating $filename\n";
        open my $fh, ">", $filename 
             or die("Could not open $filename. $!");
        print $fh $response->decoded_content;  # or whatever
        close $fh;
      } else {
        die $response->status_line;
      }
      print "\n"; 
    }

  }
}

poj