https through a proxy

[line_noise]

I'm trying to pass https requests through a squid proxy
with LWP::UserAgent and having no luck. The relevant code
looks something like this.

$PROXY_IP = "proxy.mydomain.com:8080";

$ua = LWP::UserAgent->new;
$ua->proxy(['http', 'https'], "http://$PROXY_IP/");
$URL = "https://www.somedomain.com/index.htm";
$request = HTTP::Request->new('GET', "$URL");
$response = $ua->request($request);

At which I get back a response from the proxy server that says:

The following error was encountered:
Unsupported Request Method and Protocol
Squid does not support all request methods for all access
protocols.
For example, you can not POST a Gopher request.

the last of which I find a bit insulting...
Any suggestions? Does the proxy method not know that it
needs to send a CONNECT and wait for a RC200 from the proxy
so it can build a tunnel to the actual server? What gives?

Thanks.

[igorash]

Some time ago I had this same problem. The reason for this error is that LWP don't support HTTPS over proxy properly. It establishes TCP connection to a proxy, then creates SSL connection over TCP, and then sends "CONNECT needed-host\r\n" request. With real-world proxies (that is, squid:-), correct sequence is: create TCP connection to proxy, send "CONNECT xyz\r\n", and only then establish SSL connection. I had a patch for LWP somewhere, but it must be outdated by now. Mail me (igorash at mail.od.ua) and I'll send it to you.

[line_noise]

Thanks a lot. I had a feeling the problem was along those
lines. The really insane part is that the script I'm writing
is itself a proxy, so that I can trap netscape's outgoing
requests.

[ybiC]

    Hi line_noise,
At first glance, I'd agree with good monk jeffa - sounds like your Squid proxy may not be explicitely config'd for https.   But that's just a guess.

"(code) LWP with auth proxy" is a recently-posted tidbit using LWP with proxy.   I'll tweak it for https and try at home this weekend from behind my Squid.   Will update this node with results and /msg you.   If I can find pertinant Squid config params will include them as well.
    cheers,
    Don
    striving toward Perl Adept
    (it's pronounced "why-bick")

Update:
Looks like igorash hit the nail on the head 8^)
A Google search for LWP https pointed me at Crypt::SSLeay, which is described as "OpenSSL glue that provides LWP https support.   Syntax examples at http://search.cpan.org/doc/CHAMAS/Crypt-SSLeay-0.25/SSLeay.pm

[jeffa]

First things first: I don't got no answer. :(

But maybe I can offer some more clues. My home network consists of a Linux box that acts as a gateway and uses Squid. I have another Linux box and a Windows box that use the first as their gateway. I can successfully run Netscape and IE (respective boxes) via the proxy, but not Napster, and my inner Linux box experiences your exact same problem with LWP scripts that try to communicate throught the proxy.

I actually wrote a script once and had ybiC test it out for me. It failed on my box, but not his.

It has to be a Squid configuration problem, not LWP.

Update: oops! igorash++ and ybiC++

Jeff

R-R-R--R-R-R--R-R-R--R-R-R--R-R-R--
L-L--L-L--L-L--L-L--L-L--L-L--L-L--

[strredwolf]

There's a C program called micro_proxy that handles https. Check freshmeat on it.

--
$Stalag99{"URL"}="http://stalag99.keenspace.com";

[line_noise]

Hmmm.... http requests don't have a problem getting through,
just https. Has anyone ever tried any kind of workaround
using just IO::Socket calls? I'm reluctant to rewrite LWP
if I don't have to...