Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

LWP::Useragent GET fails with https

by Anonymous Monk
on Sep 16, 2010 at 06:13 UTC ( #860319=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, Here's my code
#!/path/to/perl use strict; use LWP::UserAgent; use HTTP::Request::Common; use Crypt::SSLeay; $ENV{HTTPS_PROXY} = 'http://proxy:8080/'; $ENV{HTTPS_DEBUG} = 1; my $myurl = "https://www.redhat.com"; my $ua = new LWP::UserAgent; $ua->cookie_jar( {} ); $ua->protocols_allowed( [ 'http','https'] ); $ua->proxy(['http', 'https'], 'http://proxy:8080/'); my $page = $ua->get($myurl); die "Error $myurl\n ", $page->status_line, "\n Aborting" unless $page->is_success; print "Success", $page1->content_type, " document!\n";
It returns
Error at https://www.redhat.com 400 Bad Request Aborting at test.pl line 30.
Apparently, Its a bug: http://rt.cpan.org/Public/Bug/Display.html?id=1894 The workaround doesn't work for me. The proxy connection fails with it

Comment on LWP::Useragent GET fails with https
Select or Download Code
Re: LWP::Useragent GET fails with https
by zentara (Archbishop) on Sep 16, 2010 at 10:24 UTC
Re: LWP::Useragent GET fails with https
by wazoox (Prior) on Sep 16, 2010 at 10:42 UTC

    If you define $ENV{HTTPS_PROXY} = 'http://proxy:8080'; you must NOT declare the proxy to LWP. So the correct code is:

    #!/path/to/perl use strict; use LWP::UserAgent; use HTTP::Request::Common; use Crypt::SSLeay; $ENV{HTTPS_PROXY} = 'http://proxy:8080'; $ENV{HTTPS_DEBUG} = 1; my $myurl = "https://www.redhat.com"; my $ua = new LWP::UserAgent; $ua->cookie_jar( {} ); $ua->protocols_allowed( [ 'http','https'] ); # don't use https proxy with LWP! $ua->proxy(['http'], 'http://proxy:8080'); my $page = $ua->get($myurl); die "Error $myurl\n ", $page->status_line, "\n Aborting" unless $page->is_success; print "Success", $page->content_type, " document!\n";

    Does this work for you?

    Edit: corrected, removing the trailing slash from proxy url

      I'm still always getting "400 Bad Request" errors when I use the https proxy. If I try the code exactly as written, I get 500 errors so it's not using the https proxy at all via the environment variable.
        There was a slight mistake in my code example:
        $ENV{HTTPS_PROXY} = 'http://proxy:8080';
        Note the absence of trailing slash. It works for me, I've just got it running through my local squid proxy:
        [~/temp]$ ./test.pl SSL_connect:before/connect initialization SSL_connect:SSLv2/v3 write client hello A SSL_connect:SSLv3 read server hello A SSL_connect:SSLv3 read server certificate A SSL_connect:SSLv3 read server done A SSL_connect:SSLv3 write client key exchange A SSL_connect:SSLv3 write change cipher spec A SSL_connect:SSLv3 write finished A SSL_connect:SSLv3 flush data SSL_connect:SSLv3 read finished A Successtext/htmlcharset=ISO-8859-1 document!

        Please provide more information on your configuration. It works fine for me on Linux, running Perl 5.10.1 and LWP 5.835.

Re: LWP::Useragent GET fails with https
by danny0085 (Sexton) on Sep 20, 2012 at 15:15 UTC
    use Net::SSL (); # From Crypt-SSLeay use LWP::UserAgent; $Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL $ENV{HTTPS_PROXY} = 'http://98.142.214.160:443'; $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new('GET','https://twitter.com'); my $res = $ua->request($req); print $res->status_line;

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://860319]
Approved by planetscape
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (5)
As of 2014-09-01 22:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (17 votes), past polls