Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

Connection through a proxy

by freak
on Apr 26, 2004 at 11:21 UTC ( #348137=perlquestion: print w/replies, xml ) Need Help??
freak has asked for the wisdom of the Perl Monks concerning the following question:

What's the simplest script to connect to a website(web-server) through a proxy ?

20040426 Corion: Moved from Perl Monks Discussion to SoPW

Replies are listed 'Best First'.
Re: Connection through a proxy
by Corion (Pope) on Apr 26, 2004 at 11:28 UTC

    The simplest script is:

    # in your shell: perl -MLWP::Simple -e "get ''"

    assuming, that you have set your environment variables up correctly, that is, HTTP_PROXY=

      How to do it with Apache ?

        I'm not sure what your question is, because your one-line questions don't let out much information. So I will guess at what you possibly mean:

        Maybe your wanted the answer to the question

        What is the simplest Perl script to request a page from an Apache webserver through a proxy?

        This question is still answered by my response to your first question.

        Maybe your wanted the answer to the question

        What is the simplest Perl script to request a page from through a proxy?

        This question is still answered by my first answer to your post.

        Since I don't want to assume that you post your questions without any goal, maybe the following question is the question you really wanted an answer for:

        What is a simple Perl script that I can run as a CGI from within Apache which will request a page from another webserver through a proxy?

        For that question, you should look at how HTTP::Proxy does it, and it is not really simple. Most of the meat will be done through for the page and LWP::UserAgent or LWP::Simple to fire off the request.

        If, what I think is most likely, none of these answers are the answers to your real question that you mean but did not write, then please do read How (Not) To Ask A Question and then post your real question, together with the code you already have.

Re: Connection through a proxy
by TwistedGreen (Sexton) on Apr 26, 2004 at 12:52 UTC
    Will you need to use authentication to get through the proxy? If so, you may run into some difficulties using the usual modules. I have had to use the CPAN module Net::HTTPTunnel instead of any of the LWP::UserAgent modules to do this.

    An example:
    use Net::HTTPTunnel; use HTTP::Request; my $proxy_host = ""; my $proxy_port = "83"; my $proxy_user = "username"; my $proxy_pass = "password"; my $ht = Net::HTTPTunnel->new( 'proxy-host' => $proxy_host, 'proxy-port' => $proxy_port, 'remote-host' => '', 'remote-port' => '80', 'proxy-user' => $proxy_user, 'proxy-pass' => $proxy_pass, 'timeout' => 20); $ht or die "request timed out"; # generate an HTTP request my $req = HTTP::Request->new(GET => " +?node_id=317556"); print $ht $req->as_string; # print the request to t +he socket my $response; while (<$ht>) {$response .= $_;} # read response (the HTML +web page) from the socket close $ht; # close the socket
    The page is now stored in $response, to do with as you please.

      What is the difference with this module, and the Basic authentication that comes with LWP?

      See the credentials function in LWP::UserAgent.

      Dr. Mark Ceulemans
      Senior Consultant
      BMC, Belgium
        From what I gathered when I was working on this problem, the main difference is in the level of abstraction of the modules. LWP::UserAgent will not give you a direct socket, while Net::HTTPTunnel will return a TCP socket which is tunneled transparently through the proxy you specify. You can then write what you will to the server, such as the HTTP GET request as shown in the example.

        But if you just want to retrieve a page like I did, the critical difference was that Net::HTTPTunnel worked swimmingly with my proxy while LWP::UserAgent seemed unable to authenticate properly. Perhaps a more experienced monk can explain the underlying mechanics.

      The newer LWP::UserAgent modules accept an http_proxy in the form http://username:password@server:port/ . You can, for instance, set $ENV{HTTP_PROXY} to a string of that form and then use LWP::Simple, etc.

      If you have a Microsoft proxy server that you are going through, you will have to indicate your domain as well, so the format will be http://domain\username:password@server:port/ (careful with quoting of the backslash).

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://348137]
Approved by Corion
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (5)
As of 2018-04-23 10:01 GMT
Find Nodes?
    Voting Booth?