http://www.perlmonks.org?node_id=22449

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

To get a webpage I could do:

use LWP::Simple;
$webpage=get "http://www.perlmonks.org";

How can this be done if a proxy is present? I tried reading the HTTP and HTTP 1.1 FAQs and am lacking information/understanding on how to request pages from a proxy.

Replies are listed 'Best First'.
Re: Getting a webpage through proxy
by lhoward (Vicar) on Jul 14, 2000 at 00:58 UTC
    You can't do it with LWP::Simple, you have to use LWP::UserAgent to get proxy support.... 
    require LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->proxy('http', 'http://proxy.sn.no:8001/');
$request = HTTP::Request->new('GET', 'http://www.perlmonks.org');
$response = $ua->request($request);

    [download]
    Note that $response is not just the text of the gotten page, it is an HTTP::Response object. See the HTTP::Response docs for info on how to handle the response.
[reply]
[d/l]
      Check out -
      $ua->proxy('http', 'http://proxy.name.ie:8001/');
      And this node
[reply]
[d/l]
Re: Getting a webpage through proxy
by Abigail (Deacon) on Jul 14, 2000 at 08:53 UTC
    The manuals will tell you. man LWP::Simple tells you that you can import the UserAgent object, and tells you which method to call to load the proxy information from the environment. It also points to the LWP::UserAgent manpage which has further details on how to set which proxies.

    -- Abigail

[reply]
RE: Getting a webpage through proxy
by flyfishin (Monk) on Jul 14, 2000 at 00:54 UTC
    This node http://www.perlmonks.org/index.pl?node_id=20200 is a long discussion of this topic. As soon as I figure out how to get that to a link I will update it. Off to the FAQ I go.

    Update:
    How about this.
[reply]
Re: Getting a webpage through proxy
by eclecticIO (Beadle) on Jul 14, 2000 at 04:01 UTC
    You can use LWP::Simple if you set (and/or export) the environment variable "http_proxy='http://your.proxy:port/'."
    I think this will work for any script using the LWP modules.

    NOTE: Be careful if you're considering this for any type of CGI script. Setting the environment
    variable won't work for the typical setup where the server runs as nobody since nobody should not
    have an environement. For use in CGI scripts (if you're running Apache (and if you're not you should :) ))
    you'll have to declare the proxy explicitly as above or use something like mod_env or suEXEC.
[reply]
Re: Getting a webpage through proxy
by Anonymous Monk on Jul 14, 2000 at 05:03 UTC
    For CGI scripts, you can add

    BEGIN { $ENV{http_proxy} = "http://your.proxy:port/" };

    To the top of the script.

[reply]
RE: Getting a webpage through proxy
by Jonathan (Curate) on Jul 14, 2000 at 14:45 UTC
    I raised this same question last week. Must admit the documentation seemed vague - I ended up looking though the pm file! Anyway I got it working with something like..

    use LWP::UserAgent;
#..... usual stuff .....#
$ua = new LWP::UserAgent;
$ua->proxy(['http', 'ftp'] => 'http://your_proxy.address.com:');

my $req = new HTTP::Request 'GET',
    "http://www.perlmonks.org";

$req->proxy_authorization_basic("$user_name","$user_password");

$res = $ua->request($req);

    [download]


    Update - I've just seen that chromatic posted a virtually identical code snipet in answer to my original question.
    Never saw it at the time Doh! That will teach me to give up on the perl monks.
[reply]
[d/l]
      in reply to jonathan's posting of working code :

      Strange, i copy pasted this code, altered params where necessarry and it don't work.

      i get something like

      HTTP::Request=HASH(0x6ade884)
      HTTP::Response=HASH(0x6ca5aa0)

      as output

      must say i'm using perl/perlscript/asp on IIS 5.0

      Can anyone help me out, post a resource for this ?

      thanks,

      joris.lambrecht@pandora.be
[reply]
Re: Getting a webpage through proxy
by Anonymous Monk on Jul 14, 2000 at 17:17 UTC
    Thanks guys. I found LWP::UserAgent, this is great! I wish I knew about PerlMonks months ago. 8-)
[reply]

Back to Seekers of Perl Wisdom