Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.

https through a proxy

by line_noise (Sexton)
on May 05, 2001 at 00:35 UTC ( [id://78114] : perlquestion . print w/replies, xml ) Need Help??

line_noise has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to pass https requests through a squid proxy
with LWP::UserAgent and having no luck. The relevant code
looks something like this.

$PROXY_IP = ""; $ua = LWP::UserAgent->new; $ua->proxy(['http', 'https'], "http://$PROXY_IP/"); $URL = ""; $request = HTTP::Request->new('GET', "$URL"); $response = $ua->request($request);
At which I get back a response from the proxy server that says:

The following error was encountered: Unsupported Request Method and Protocol Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

the last of which I find a bit insulting...
Any suggestions? Does the proxy method not know that it
needs to send a CONNECT and wait for a RC200 from the proxy
so it can build a tunnel to the actual server? What gives?


Replies are listed 'Best First'.
Re: https through a proxy
by igorash (Initiate) on May 05, 2001 at 01:10 UTC
    Some time ago I had this same problem. The reason for this error is that LWP don't support HTTPS over proxy properly. It establishes TCP connection to a proxy, then creates SSL connection over TCP, and then sends "CONNECT needed-host\r\n" request. With real-world proxies (that is, squid:-), correct sequence is: create TCP connection to proxy, send "CONNECT xyz\r\n", and only then establish SSL connection. I had a patch for LWP somewhere, but it must be outdated by now. Mail me (igorash at and I'll send it to you.
      Thanks a lot. I had a feeling the problem was along those
      lines. The really insane part is that the script I'm writing
      is itself a proxy, so that I can trap netscape's outgoing
Re: https through a proxy (igorash++, Crypt::SSLeay)
by ybiC (Prior) on May 05, 2001 at 01:12 UTC
        Hi line_noise,
    At first glance, I'd agree with good monk jeffa - sounds like your Squid proxy may not be explicitely config'd for https.   But that's just a guess.

    "(code) LWP with auth proxy" is a recently-posted tidbit using LWP with proxy.   I'll tweak it for https and try at home this weekend from behind my Squid.   Will update this node with results and /msg you.   If I can find pertinant Squid config params will include them as well.
        striving toward Perl Adept
        (it's pronounced "why-bick")

    Looks like igorash hit the nail on the head 8^)
    A Google search for LWP https pointed me at Crypt::SSLeay, which is described as "OpenSSL glue that provides LWP https support.   Syntax examples at

(jeffa) Re: https through a proxy
by jeffa (Bishop) on May 05, 2001 at 00:47 UTC
    First things first: I don't got no answer. :(

    But maybe I can offer some more clues. My home network consists of a Linux box that acts as a gateway and uses Squid. I have another Linux box and a Windows box that use the first as their gateway. I can successfully run Netscape and IE (respective boxes) via the proxy, but not Napster, and my inner Linux box experiences your exact same problem with LWP scripts that try to communicate throught the proxy.

    I actually wrote a script once and had ybiC test it out for me. It failed on my box, but not his.

    It has to be a Squid configuration problem, not LWP.

    Update: oops! igorash++ and ybiC++


Re: https through a proxy
by strredwolf (Chaplain) on May 05, 2001 at 03:51 UTC
    There's a C program called micro_proxy that handles https. Check freshmeat on it.


Re: https through a proxy
by line_noise (Sexton) on May 05, 2001 at 01:01 UTC
    Hmmm.... http requests don't have a problem getting through,
    just https. Has anyone ever tried any kind of workaround
    using just IO::Socket calls? I'm reluctant to rewrite LWP
    if I don't have to...