Yet another LWP question

[Jonathan]

Like many others I need to download a page from the web. The problem (I think I have) is that I am at work behind our firewall and to connect to the net I have to enter my username/password. I think this is why all the example scripts are failing.

Is there any way I can automate this? Is there a module that will pass these as parameters?
I've looked at the pm files and nothing suggested itself.

help!

[flyfishin]

With our firewall we essentially had to fake out the firewall and tell it that we are using a Mozilla agent to get out, and also gave it the proxy. To get outside our file wall this was added to our code



$ua->agent("Mozilla/4.51 [en] (X11; I; Linux 2.2.14-15mdk i686)");
$ua->proxy('http', 'http://ourproxy:8080');

[lhoward]

The crux of your problem is understanding how your firewall/proxy works. You need to come to an understanding of how your firewall authentication works, then you can address how to deal with it in perl. Exactly how does the login you have to do to browse the web work? Does your web browser have to have any proxy settings?

There are proxy methods in LWP::UserAgent to support proxying and other methods for HTTP authentication; either of those might fit the bill depending on how your firewall authentication works.

[gronkulator]

it depends - if you are going through an http proxy, you may have to send the proxy-authorization header as part of the http request object. You'll probably have to use the "basic" auth type, and use MIME::Base64 to encode the string "user:password" for your credentials. A quick way to cheat past this is to sniff a session from your browser and cut & paste the proxy auth header into your perl script as part of the http request headers.

[barndoor]

You could try using the credentials() method of UserAgent to set a username and password for specific URL ranges. I've not tried it myself but this defines HTTP::Headers which are used by the HTTP::Request method. I've only really just looked this up so if anyone can give an example they've done better than me. You can find out more on this in 'Nutshell'

[ybiC]

If it doesn't have to be done with Perl, GNU wget supports authenticated proxy and SOCKS. Unix, Win95, and DOS ports.

I've had good luck with it on Debian, anyway.

    cheers,
    ybiC

[chromatic]

A quick glance at `perldoc lwpcook` gives the following:

use LWP::UserAgent;

$ua = new LWP::UserAgent;
$ua->proxy(['http', 'ftp'] => 'http://proxy.myorg.com');

$req = new HTTP::Request 'GET',"http://www.perl.com";
$req->proxy_authorization_basic("proxy_user", "proxy_password");

$res = $ua->request($req);
print $res->content if $res->is_success;

If your OS supports it, you might also be able to set HTTP_PROXY and FTP_PROXY environment variables. But I'd do it the LWP Cookbook way.