Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

LWP GET of https URL: 400 Bad Request

by {}think (Acolyte)
on Sep 24, 2012 at 14:42 UTC ( #995402=perlquestion: print w/ replies, xml ) Need Help??
{}think has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to use LWP to GET a page using the https protocol, and for the life of me, no matter what I try, I get a "400 Bad request" response from the server (or at least I assume it's from the server). In the source below, I use LWP to first GET https://mail.google.com, which fails with a rc 400, then I GET http://mail.google.com, which gets OK. I can read both addresses using my browser. I am behind a firewall, but I am able to get through it OK for the http protocol, and I think I'm gettign through with https as well, based on the rc 400.

This topic has been brought up in the past, and I've gathered snippets from all the threads I could find, to no avail. Any suggestions? How can I go to the next level of debugging (obviously, I can use -d)? Is there some way to look at the request that was sent to see if I can make out how my request is bad? I'm at my wit's end on this one.


Source

#!/usr/bin/perl -w use strict; use warnings; use Data::Dumper; use LWP::UserAgent; use Crypt::SSLeay; use Net::SSL (); # From Crypt-SSLeay $Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL (? +) my $proxyinfo='http://bday:adsfa3@144.92.221.19'; $ENV{HTTPS_VERSION} = '2'; $ENV{HTTP_PROXY} = $proxyinfo; $ENV{HTTPS_PROXY} = $proxyinfo; $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; $ENV{PERL_LWP_ENV_PROXY} = $proxyinfo; my $ua; my $url = 'https://mail.google.com'; print "--------------------------- Test #1: $url --------------------- +------\n"; $ua = LWP::UserAgent->new(show_progress=>1); $ua->cookie_jar( {} ); $ua->protocols_allowed( [ 'http','https'] ); $ua->env_proxy; # $ua->proxy(['https'], $proxyinfo); my $req = HTTP::Request->new('GET',$url); print Dumper($req); my $res = $ua->request($req); print $res->status_line; print $res->as_string; #$url = 'https://www.helsinki.fi/'; $url = 'http://mail.google.com/'; print "--------------------------- Test #2: $url --------------------- +------\n"; #my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 }); #$ua->protocols_allowed( [ 'http', 'https'] ); #bd added - may be unne +cessary $res = $ua->get( $url ); $res->is_success or die "Failed to GET '$url': ", $res->status_line; print $res->as_string; print "--------------------------- SSL in \%INC ---------------------- +-----\n"; print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;

Results

Name "Net::HTTPS::SSL_SOCKET_CLASS" used only once: possible typo at . +/lwpssltest.pl line 8. --------------------------- Test #1: https://mail.google.com --------- +------------------ $VAR1 = bless( { '_content' => '', '_uri' => bless( do{\(my $o = 'https://mail.google.co +m')}, 'URI::https' ), '_headers' => bless( {}, 'HTTP::Headers' ), '_method' => 'GET' }, 'HTTP::Request' ); ** GET https://mail.google.com ==> 400 Bad Request (1s) 400 Bad RequestHTTP/1.1 400 Bad Request Cache-Control: no-cache Connection: close Pragma: no-cache Content-Length: 691 Content-Type: text/html; charset=utf-8 Client-Date: Mon, 24 Sep 2012 14:31:14 GMT Client-Peer: 144.72.231.19:80 Client-Response-Num: 1 Proxy-Connection: close Title: Request Error <HTML><HEAD> <TITLE>Request Error</TITLE> </HEAD> <BODY> <FONT face="Helvetica"> <big><strong></strong></big><BR> </FONT> <blockquote> <TABLE border=0 cellPadding=1 width="80%"> <TR><TD> <FONT face="Helvetica"> <big>Request Error (invalid_request)</big> <BR> <BR> </FONT> </TD></TR> <TR><TD> <FONT face="Helvetica"> Your request could not be processed. Request could not be handled </FONT> </TD></TR> <TR><TD> <FONT face="Helvetica"> This could be caused by a misconfiguration, or possibly a malformed re +quest. </FONT> </TD></TR> <TR><TD> <FONT face="Helvetica" SIZE=2> <BR> For assistance, contact your network support team. </FONT> </TD></TR> </TABLE> </blockquote> </FONT> </BODY></HTML> --------------------------- Test #2: http://mail.google.com/ --------- +------------------ ** GET http://mail.google.com/ ==> 200 OK HTTP/1.1 200 OK Cache-Control: private, max-age=604800 Connection: close Date: Mon, 24 Sep 2012 14:31:14 GMT Server: GSE Content-Length: 232 Content-Type: text/html; charset=ISO-8859-1 Expires: Mon, 24 Sep 2012 14:31:14 GMT Client-Date: Mon, 24 Sep 2012 14:31:14 GMT Client-Peer: 144.72.231.19:80 Client-Response-Num: 1 Refresh: 0;URL=http://mail.google.com/mail/ X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block <html><head><meta http-equiv="Refresh" content="0;URL=http://mail.goog +le.com/mail/" /></head><body><script type="text/javascript" language= +"javascript"><!-- location.replace("http://mail.google.com/mail/") --></script></body></html> --------------------------- SSL in %INC --------------------------- Crypt/SSLeay.pm Crypt/SSLeay/X509.pm Net/SSL.pm

Versions

SunOS 5.10 Perl v5.10.1. LWP 6.04 Crypt::SSLeay 0.57 Net::SSL 2.84 LWP::UserAgent 6.04 LWP::Protocol::https 6.03
{}think; #Think outside of the brackets

Comment on LWP GET of https URL: 400 Bad Request
Select or Download Code
Re: LWP GET of https URL: 400 Bad Request
by Corion (Pope) on Sep 24, 2012 at 14:46 UTC

    Are you sure that the 400 Error comes from Google? I would suspect your company proxy, because I think that Crypt::SSLeay likes to take its parameters out of %ENV at module load time. I would try putting the environment setup in a BEGIN block before loading Crypt::SSLeay and related modules.

      I got through! Thank you for your assistance!
      {}think; #Think outside of the brackets
        I am facing the same issue. Could you please share your solution?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://995402]
Approved by Kenosis
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (6)
As of 2015-07-04 04:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (57 votes), past polls