LWP GET of https URL: 400 Bad Request

{}think has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to use LWP to GET a page using the https protocol, and for the life of me, no matter what I try, I get a "400 Bad request" response from the server (or at least I assume it's from the server). In the source below, I use LWP to first GET https://mail.google.com, which fails with a rc 400, then I GET http://mail.google.com, which gets OK. I can read both addresses using my browser. I am behind a firewall, but I am able to get through it OK for the http protocol, and I think I'm gettign through with https as well, based on the rc 400.

This topic has been brought up in the past, and I've gathered snippets from all the threads I could find, to no avail. Any suggestions? How can I go to the next level of debugging (obviously, I can use -d)? Is there some way to look at the request that was sent to see if I can make out how my request is bad? I'm at my wit's end on this one.

Source


#!/usr/bin/perl -w
use strict;
use warnings;
use Data::Dumper;
use LWP::UserAgent;
use Crypt::SSLeay;
use Net::SSL (); # From Crypt-SSLeay
$Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL (?
+)
my $proxyinfo='http://bday:adsfa3@144.92.221.19';

$ENV{HTTPS_VERSION} = '2';
$ENV{HTTP_PROXY} =  $proxyinfo;
$ENV{HTTPS_PROXY} =  $proxyinfo;
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{PERL_LWP_ENV_PROXY} =  $proxyinfo;

my $ua;
my $url = 'https://mail.google.com';

print "--------------------------- Test #1: $url ---------------------
+------\n";
$ua = LWP::UserAgent->new(show_progress=>1);
$ua->cookie_jar( {} );
$ua->protocols_allowed( [ 'http','https'] );
$ua->env_proxy;
# $ua->proxy(['https'], $proxyinfo);
my $req = HTTP::Request->new('GET',$url);
print Dumper($req);
my $res = $ua->request($req);
print $res->status_line;
print $res->as_string;

#$url = 'https://www.helsinki.fi/';
$url = 'http://mail.google.com/';
print "--------------------------- Test #2: $url ---------------------
+------\n";
#my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 });
#$ua->protocols_allowed( [ 'http', 'https'] ); #bd added - may be unne
+cessary
$res = $ua->get( $url );
$res->is_success or
    die "Failed to GET '$url': ", $res->status_line;
print $res->as_string;

print "--------------------------- SSL in \%INC ----------------------
+-----\n";
print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;
[download]

Results

Name "Net::HTTPS::SSL_SOCKET_CLASS" used only once: possible typo at .
+/lwpssltest.pl line 8.
--------------------------- Test #1: https://mail.google.com ---------
+------------------
$VAR1 = bless( {
                 '_content' => '',
                 '_uri' => bless( do{\(my $o = 'https://mail.google.co
+m')}, 'URI::https' ),
                 '_headers' => bless( {}, 'HTTP::Headers' ),
                 '_method' => 'GET'
               }, 'HTTP::Request' );
** GET https://mail.google.com ==> 400 Bad Request (1s)
400 Bad RequestHTTP/1.1 400 Bad Request
Cache-Control: no-cache
Connection: close
Pragma: no-cache
Content-Length: 691
Content-Type: text/html; charset=utf-8
Client-Date: Mon, 24 Sep 2012 14:31:14 GMT
Client-Peer: 144.72.231.19:80
Client-Response-Num: 1
Proxy-Connection: close
Title: Request Error

<HTML><HEAD>
<TITLE>Request Error</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Request Error (invalid_request)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your request could not be processed. Request could not be handled
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
This could be caused by a misconfiguration, or possibly a malformed re
+quest.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
--------------------------- Test #2: http://mail.google.com/ ---------
+------------------
** GET http://mail.google.com/ ==> 200 OK
HTTP/1.1 200 OK
Cache-Control: private, max-age=604800
Connection: close
Date: Mon, 24 Sep 2012 14:31:14 GMT
Server: GSE
Content-Length: 232
Content-Type: text/html; charset=ISO-8859-1
Expires: Mon, 24 Sep 2012 14:31:14 GMT
Client-Date: Mon, 24 Sep 2012 14:31:14 GMT
Client-Peer: 144.72.231.19:80
Client-Response-Num: 1
Refresh: 0;URL=http://mail.google.com/mail/
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block

<html><head><meta http-equiv="Refresh" content="0;URL=http://mail.goog
+le.com/mail/" /></head><body><script type="text/javascript" language=
+"javascript"><!--
location.replace("http://mail.google.com/mail/")
--></script></body></html>
--------------------------- SSL in %INC ---------------------------
Crypt/SSLeay.pm
Crypt/SSLeay/X509.pm
Net/SSL.pm
[download]

Versions


SunOS 5.10
Perl v5.10.1.  
LWP 6.04
Crypt::SSLeay 0.57
Net::SSL 2.84
LWP::UserAgent 6.04
LWP::Protocol::https 6.03
[download]

{}think; #Think outside of the brackets

Comment on LWP GET of https URL: 400 Bad Request Select or Download Code

Replies are listed 'Best First'.
Re: LWP GET of https URL: 400 Bad Request by Corion (Patriarch) on Sep 24, 2012 at 14:46 UTC
Are you sure that the 400 Error comes from Google? I would suspect your company proxy, because I think that Crypt::SSLeay likes to take its parameters out of %ENV at module load time. I would try putting the environment setup in a `BEGIN` block before loading Crypt::SSLeay and related modules.	[reply] [d/l]
Re^2: LWP GET of https URL: 400 Bad Request by {}think (Sexton) on Sep 26, 2012 at 21:57 UTC
I got through! Thank you for your assistance! {}think; #Think outside of the brackets	[reply]
Re^3: LWP GET of https URL: 400 Bad Request by njaggi (Initiate) on Oct 25, 2012 at 23:57 UTC
I am facing the same issue. Could you please share your solution?	[reply]

Back to Seekers of Perl Wisdom