Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

LWP::Useragent GET fails with https

by Anonymous Monk
on Sep 16, 2010 at 06:13 UTC ( #860319=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, Here's my code
#!/path/to/perl use strict; use LWP::UserAgent; use HTTP::Request::Common; use Crypt::SSLeay; $ENV{HTTPS_PROXY} = 'http://proxy:8080/'; $ENV{HTTPS_DEBUG} = 1; my $myurl = "https://www.redhat.com"; my $ua = new LWP::UserAgent; $ua->cookie_jar( {} ); $ua->protocols_allowed( [ 'http','https'] ); $ua->proxy(['http', 'https'], 'http://proxy:8080/'); my $page = $ua->get($myurl); die "Error $myurl\n ", $page->status_line, "\n Aborting" unless $page->is_success; print "Success", $page1->content_type, " document!\n";
It returns
Error at https://www.redhat.com 400 Bad Request Aborting at test.pl line 30.
Apparently, Its a bug: http://rt.cpan.org/Public/Bug/Display.html?id=1894 The workaround doesn't work for me. The proxy connection fails with it

Comment on LWP::Useragent GET fails with https
Select or Download Code
Replies are listed 'Best First'.
Re: LWP::Useragent GET fails with https
by wazoox (Prior) on Sep 16, 2010 at 10:42 UTC

    If you define $ENV{HTTPS_PROXY} = 'http://proxy:8080'; you must NOT declare the proxy to LWP. So the correct code is:

    #!/path/to/perl use strict; use LWP::UserAgent; use HTTP::Request::Common; use Crypt::SSLeay; $ENV{HTTPS_PROXY} = 'http://proxy:8080'; $ENV{HTTPS_DEBUG} = 1; my $myurl = "https://www.redhat.com"; my $ua = new LWP::UserAgent; $ua->cookie_jar( {} ); $ua->protocols_allowed( [ 'http','https'] ); # don't use https proxy with LWP! $ua->proxy(['http'], 'http://proxy:8080'); my $page = $ua->get($myurl); die "Error $myurl\n ", $page->status_line, "\n Aborting" unless $page->is_success; print "Success", $page->content_type, " document!\n";

    Does this work for you?

    Edit: corrected, removing the trailing slash from proxy url

      I'm still always getting "400 Bad Request" errors when I use the https proxy. If I try the code exactly as written, I get 500 errors so it's not using the https proxy at all via the environment variable.
        There was a slight mistake in my code example:
        $ENV{HTTPS_PROXY} = 'http://proxy:8080';
        Note the absence of trailing slash. It works for me, I've just got it running through my local squid proxy:
        [~/temp]$ ./test.pl SSL_connect:before/connect initialization SSL_connect:SSLv2/v3 write client hello A SSL_connect:SSLv3 read server hello A SSL_connect:SSLv3 read server certificate A SSL_connect:SSLv3 read server done A SSL_connect:SSLv3 write client key exchange A SSL_connect:SSLv3 write change cipher spec A SSL_connect:SSLv3 write finished A SSL_connect:SSLv3 flush data SSL_connect:SSLv3 read finished A Successtext/htmlcharset=ISO-8859-1 document!

        Please provide more information on your configuration. It works fine for me on Linux, running Perl 5.10.1 and LWP 5.835.

Re: LWP::Useragent GET fails with https
by zentara (Archbishop) on Sep 16, 2010 at 10:24 UTC
Re: LWP::Useragent GET fails with https
by danny0085 (Sexton) on Sep 20, 2012 at 15:15 UTC
    use Net::SSL (); # From Crypt-SSLeay use LWP::UserAgent; $Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL $ENV{HTTPS_PROXY} = 'http://98.142.214.160:443'; $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new('GET','https://twitter.com'); my $res = $ua->request($req); print $res->status_line;

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://860319]
Approved by planetscape
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (14)
As of 2015-07-07 22:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (93 votes), past polls