Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

LWP GET of https URL: 400 Bad Request

by {}think (Acolyte)
on Sep 24, 2012 at 14:42 UTC ( #995402=perlquestion: print w/ replies, xml ) Need Help??
{}think has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to use LWP to GET a page using the https protocol, and for the life of me, no matter what I try, I get a "400 Bad request" response from the server (or at least I assume it's from the server). In the source below, I use LWP to first GET https://mail.google.com, which fails with a rc 400, then I GET http://mail.google.com, which gets OK. I can read both addresses using my browser. I am behind a firewall, but I am able to get through it OK for the http protocol, and I think I'm gettign through with https as well, based on the rc 400.

This topic has been brought up in the past, and I've gathered snippets from all the threads I could find, to no avail. Any suggestions? How can I go to the next level of debugging (obviously, I can use -d)? Is there some way to look at the request that was sent to see if I can make out how my request is bad? I'm at my wit's end on this one.


Source

#!/usr/bin/perl -w use strict; use warnings; use Data::Dumper; use LWP::UserAgent; use Crypt::SSLeay; use Net::SSL (); # From Crypt-SSLeay $Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL (? +) my $proxyinfo='http://bday:adsfa3@144.92.221.19'; $ENV{HTTPS_VERSION} = '2'; $ENV{HTTP_PROXY} = $proxyinfo; $ENV{HTTPS_PROXY} = $proxyinfo; $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; $ENV{PERL_LWP_ENV_PROXY} = $proxyinfo; my $ua; my $url = 'https://mail.google.com'; print "--------------------------- Test #1: $url --------------------- +------\n"; $ua = LWP::UserAgent->new(show_progress=>1); $ua->cookie_jar( {} ); $ua->protocols_allowed( [ 'http','https'] ); $ua->env_proxy; # $ua->proxy(['https'], $proxyinfo); my $req = HTTP::Request->new('GET',$url); print Dumper($req); my $res = $ua->request($req); print $res->status_line; print $res->as_string; #$url = 'https://www.helsinki.fi/'; $url = 'http://mail.google.com/'; print "--------------------------- Test #2: $url --------------------- +------\n"; #my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 }); #$ua->protocols_allowed( [ 'http', 'https'] ); #bd added - may be unne +cessary $res = $ua->get( $url ); $res->is_success or die "Failed to GET '$url': ", $res->status_line; print $res->as_string; print "--------------------------- SSL in \%INC ---------------------- +-----\n"; print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;

Results

Name "Net::HTTPS::SSL_SOCKET_CLASS" used only once: possible typo at . +/lwpssltest.pl line 8. --------------------------- Test #1: https://mail.google.com --------- +------------------ $VAR1 = bless( { '_content' => '', '_uri' => bless( do{\(my $o = 'https://mail.google.co +m')}, 'URI::https' ), '_headers' => bless( {}, 'HTTP::Headers' ), '_method' => 'GET' }, 'HTTP::Request' ); ** GET https://mail.google.com ==> 400 Bad Request (1s) 400 Bad RequestHTTP/1.1 400 Bad Request Cache-Control: no-cache Connection: close Pragma: no-cache Content-Length: 691 Content-Type: text/html; charset=utf-8 Client-Date: Mon, 24 Sep 2012 14:31:14 GMT Client-Peer: 144.72.231.19:80 Client-Response-Num: 1 Proxy-Connection: close Title: Request Error <HTML><HEAD> <TITLE>Request Error</TITLE> </HEAD> <BODY> <FONT face="Helvetica"> <big><strong></strong></big><BR> </FONT> <blockquote> <TABLE border=0 cellPadding=1 width="80%"> <TR><TD> <FONT face="Helvetica"> <big>Request Error (invalid_request)</big> <BR> <BR> </FONT> </TD></TR> <TR><TD> <FONT face="Helvetica"> Your request could not be processed. Request could not be handled </FONT> </TD></TR> <TR><TD> <FONT face="Helvetica"> This could be caused by a misconfiguration, or possibly a malformed re +quest. </FONT> </TD></TR> <TR><TD> <FONT face="Helvetica" SIZE=2> <BR> For assistance, contact your network support team. </FONT> </TD></TR> </TABLE> </blockquote> </FONT> </BODY></HTML> --------------------------- Test #2: http://mail.google.com/ --------- +------------------ ** GET http://mail.google.com/ ==> 200 OK HTTP/1.1 200 OK Cache-Control: private, max-age=604800 Connection: close Date: Mon, 24 Sep 2012 14:31:14 GMT Server: GSE Content-Length: 232 Content-Type: text/html; charset=ISO-8859-1 Expires: Mon, 24 Sep 2012 14:31:14 GMT Client-Date: Mon, 24 Sep 2012 14:31:14 GMT Client-Peer: 144.72.231.19:80 Client-Response-Num: 1 Refresh: 0;URL=http://mail.google.com/mail/ X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block <html><head><meta http-equiv="Refresh" content="0;URL=http://mail.goog +le.com/mail/" /></head><body><script type="text/javascript" language= +"javascript"><!-- location.replace("http://mail.google.com/mail/") --></script></body></html> --------------------------- SSL in %INC --------------------------- Crypt/SSLeay.pm Crypt/SSLeay/X509.pm Net/SSL.pm

Versions

SunOS 5.10 Perl v5.10.1. LWP 6.04 Crypt::SSLeay 0.57 Net::SSL 2.84 LWP::UserAgent 6.04 LWP::Protocol::https 6.03
{}think; #Think outside of the brackets

Comment on LWP GET of https URL: 400 Bad Request
Select or Download Code
Re: LWP GET of https URL: 400 Bad Request
by Corion (Pope) on Sep 24, 2012 at 14:46 UTC

    Are you sure that the 400 Error comes from Google? I would suspect your company proxy, because I think that Crypt::SSLeay likes to take its parameters out of %ENV at module load time. I would try putting the environment setup in a BEGIN block before loading Crypt::SSLeay and related modules.

      I got through! Thank you for your assistance!
      {}think; #Think outside of the brackets
        I am facing the same issue. Could you please share your solution?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://995402]
Approved by Kenosis
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (11)
As of 2014-07-30 12:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (231 votes), past polls