WWW::Mechanize and proxy with username/password

[perlhack]

Hello,

I am trying to get WWW::Mechanize to run while at work. To get out to the internet from our web browser we use a proxy along with a username/password, ie. the first time I bring up my web browser to an external web address I enter my username and password and the proxy lets me out to external internet addresses.

In the past I would use LWP::UserAgent to get an external web page with the following script:

#!/usr/local/bin/perl

use LWP::UserAgent;
use URI;
use Getopt::Long;

GetOptions(qw(proxy));

$user = "myusername";
$passwd = "mypassword";

foreach $url2get (@ARGV) { 
  $url = new URI "http://$url2get";
  $ua = new LWP::UserAgent;
  $req = new HTTP::Request GET => $url;

  if($opt_proxy) {
   $ua->agent("Mozilla/4.05");  # pretend I am a Netscape browser
   $ua->proxy(['http'], 'http://wwwproxy.mycompany.com:1080');
   $req->proxy_authorization_basic($user,$passwd);
  }
  
  $res = $ua->request($req);
  
  # check the outcome
  if ($res->is_success) {
    print $res->content;
  } 
  else {
    print "Error: " . $res->code . " " . $res->message;
  }
}

Although LWP::UserAgent says it can handle a proxy I couldn't figure out a way to get it to work with a proxy that expects a username and password and somehow found in a web search something similar to how the above scripts works.

Ok, now I am trying to use WWW::Mechanize and it says to handle a proxy it relies or leverages on LWP::UserAgent for this. I can replace LWP::UserAgent with WWW::Mechanize in the above script to fetch a web page fine but the other aspects of WWW::Mechanize such as "following" a link, "click", etc. will get denied by the proxy as they dont have the URI and HTTP::Request stuff happening to them.

Has anyone ever gotten LWP::UserAgent to work with a proxy with username/password without resorting to the use of URI and HTTP::Request like above as if I can do this I then think WWW::Mechanize would work similarly?

Thanks, perlhack

[Anonymous Monk]

It's funny because I was trying to get this to work yesterday. Here's my code sample that finally did it:

my $ua = LWP::UserAgent->new;

$proxy = sprintf("http://%s:%s",$server,$port);
$ENV{HTTP_PROXY} = $proxy;

my $request = HTTP::request->new('GET', $webpage);
$ua->proxy( http => $proxy );
my $response = $ua->request($request);
if ( !$response->is_success ) {
  #FAILED
}

my $html = $response->content;

Hope this helps, Michael

[Thelonius]

If what Corion suggested doesn't work, you could try this:

if ($opt_proxy) {
  my $h = "Proxy_Authorization";
  my $req = new HTTP::Request;
  $req->proxy_authorization_basic($user, $passwd);
  $WWW::Mechanize::headers{$h} = $req->header($h);
}
# Then the rest of your program

[tilly]

What Corion suggested does work.

The biggest trick is how you encode names and passwords that include special characters like /, : and @. The answer is to URI encode them.

See Put name and password in URLs for details.

[perlhack]

Ok, everyone here is the latest.

What Corion suggested did work, ie. I could put my username and password in front of the web proxy such as:
'http://myusername:[e-mail]:1080'

But I also discovered some verbage at my work about making sure if using scripting and not a web browser to access the internet to make sure the username and password were base64 encoded and not just plug them in as is?

What Thelonius suggested also worked.

Either method seems to work and I guess the only question that remains is in either method is one any more secure than the other and if there is a way to base64 encode within these schemes.

Does anyone know if there is anyway to actually see what headers are getting sent from the perl script?

Thanks for all the help!

[tcf22]

I don't believe it is possible to authenticate to a proxy with just LWP::UserAgent. However in the WWW:Mechanize docs it says

If it's possible with LWP::UserAgent, then yes. WWW::Mechanize is a subclass of LWP::UserAgent, so all the wondrous magic of that class is inherited. .

So you should be able to use the same method(using HTTP::Request) to authenticate.

Update:May also want to try using the LWP::Authen modules.

[perlhack]

Michael,
I need to authenticate with a username and password with my proxy. I think your code does not have this part which is my problem.

tcf22,
Yes I saw the comment in WWW::Mechanize about the subclass of LWP::UserAgent but if LWP::UserAgent doesn't really support the username/password part of the proxy directly then I dont know if it will work with WWW::Mechanize.

I was able to take my code and replace LWP::UserAgent with WWW::Mechanize to do a get or fetch of a web page and that worked ok. But the other "Mechanize" capabilities of WWW::Mechanize dont seem to remember or inherit or know how to pass my username and password with its requests to the web via the proxy server.

There must be a way but I cant seem to figure it out and haven't been able to find an example WWW::Mechanize script that has proxy with username/password code shown. I guess that is why I am seeking some wisdom.

Thanks.

[Corion]

At work, I have set $ENV{HTTP_PROXY} to http://login:password@proxyserver, and it magically works :

use strict;
use WWW::Mechanize;
my $agent = WWW::Mechanize->new();
$agent->env_proxy();
$agent->get('http://www.google.com');
print $agent->content;

perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ;    # The  
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new   #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' #  web

[Thelonius]

Either method seems to work and I guess the only question that remains is in either method is one any more secure than the other and if there is a way to base64 encode within these schemes.

Does anyone know if there is anyway to actually see what headers are getting sent from the perl script?

This will show you pretty much everything that goes on:

LWP::Debug::level('+');

Or you might just want LWP::Debug::level('+conns');

My method definitely Base64-encodes the username and password, but that doesn't make it more secure, really. It's just done so there are no special characters.

[petdance]

Note that Mechanize now has a success() method so you don't have to capture the response from get().

$agent->get( $url );
$agent->success or die "Couldn't get $url: ", $agent->res->status_line
+;

xoxo,
Andy