Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

HTTPS returning 400 from Linux but 200 OK from Windows

by Random_Walk (Prior)
on Jan 27, 2017 at 08:35 UTC ( #1180446=perlquestion: print w/replies, xml ) Need Help??
Random_Walk has asked for the wisdom of the Perl Monks concerning the following question:

Most esteemed monks,

I have a script pulling back CMDB info from service now using REST API. works a treat on my windows laptop where developed. Move it to Linux and I get a 400 back. I have cut things all the way back and found it is not REST issue, but even a basic LWP get is failing. Here is my test script that work on win, fails on lin.

#!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; my $ua = LWP::UserAgent->new; $ua->timeout(10); $ua->proxy('https', $ENV{HTTPS_PROXY}); my $response = $ua->get('https://google.com/'); if ($response->is_success) { print $response->decoded_content; # or whatever } else { die $response->status_line; }
Here is the curl test
myaccount@linuxhost:~> curl https://www.google.com <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset +=utf-8"> <TITLE>302 Moved</TITLE></HEAD><BODY> <H1>302 Moved</H1> The document has moved <A HREF="https://www.google.se/?gfe_rd=cr&amp;ei=XwaLWMjfB6Sr8weh06_QD +Q">here</A>. </BODY></HTML>
My HTTPS_PROXY is set, and I can pull back https://google.com with curl. What magic do I miss?

Cheers,
R.

Pereant, qui ante nos nostra dixerunt!

Replies are listed 'Best First'.
Re: HTTPS returning 400 from Linux but 200 OK from Windows
by Corion (Pope) on Jan 27, 2017 at 09:10 UTC

    Consider looking at the actual data that gets exchanged via LWP::ConsoleLogger maybe.

    I assume that your proxy wants a username+password, which you don't give it. Maybe curl is better or uses some different credentials there.

      I found a 5.18 perl on a network adjacent SuSE 12sp1 server

      me@linux:/home/rpollard> export HTTPS_PROXY='http://user:pass@proxy:80 +80' me@linux:/home/rpollard> echo $HTTPS_PROXY http://user:pass@proxy:8080 me@linux:/home/rpollard> curl https://www.google.com <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset +=utf-8"> <TITLE>302 Moved</TITLE></HEAD><BODY> <H1>302 Moved</H1> The document has moved <A HREF="https://www.google.se/?gfe_rd=cr&amp;ei=6T-MWJGBLsiq8weZ_YCIA +Q">here</A>. </BODY></HTML> me@linux:/home/rpollard> perl GetTest.pl 400 Bad Request at GetTest.pl line 16. me@linux:/home/rpollard> unset HTTPS_PROXY me@linux:/home/rpollard> echo $HTTPS_PROXY me@linux:/home/rpollard> curl https://www.google.com curl: (6) Could not resolve host: www.google.com me@linux:/home/rpollard> perl GetTest.pl 500 Can't connect to google.com:443 (Bad hostname) at GetTest.pl line +16. me@linux:/home/rpollard> perl -v This is perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-li +nux-thread-multi Copyright 1987-2013, Larry Wall Perl may be copied only under the terms of either the Artistic License + or the GNU General Public License, which may be found in the Perl 5 source ki +t. Complete documentation for Perl, including FAQ lists, should be found +on this system using "man perl" or "perldoc perl". If you have access to + the Internet, point your browser at http://www.perl.org/, the Perl Home Pa +ge.
      That is using the same test script as in the root node. I think this shows both curl and perl are using the same HTTPS_PROXY here. Now I have a more modern Perl, I will install LWP::ConsoleLogger and see what that tells me.

      Cheers,
      R.

      Pereant, qui ante nos nostra dixerunt!

        Server is all locked down, no gcc/cc etc and seos won't even let me install gcc, so I can't cpan LWP::ConsoleLogger. Now I can either go through the full change cycle to get a test server, or use curl in my scripts to pull back the data form the REST API I am after, then parse it from there. Neither choice appeals


        Cheers,
        R.

        Pereant, qui ante nos nostra dixerunt!

      Hi Corion,

      I have HTTP_PROXY and HTTPS_PROXY set with http://user:pass@proxyhost and these are used by curl so must be correct.

      I tried to install lwp::ConsoleLogger with CPAN, and it failed as my Perl is too old (5.10). I suspect this may be the cause, so I am having a quick look around for a server in the same data center with a newer Perl to test if that is the critical difference. The Perl on my lappy that works is 5.22.1.

      Cheers,
      R.

      Pereant, qui ante nos nostra dixerunt!

      I have LWP::ConsoleLogger installed after much struggle to get it built within the restrictions of the client site. Sadly it has brought me no new enlightenment. My only though now is that Perl does not like my https_proxy being an http url.

      # curl https://google.com/ <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset +=utf-8"> <TITLE>302 Moved</TITLE></HEAD><BODY> <H1>302 Moved</H1> The document has moved <A HREF="https://www.google.se/?gfe_rd=cr&amp;ei=IvWhWPTvI4b67gSTn7ygA +g">here</A>. </BODY></HTML> # perl test.pl GET https://google.com/ .-----------------+--------------------. | Request Header | Value | +-----------------+--------------------+ | Accept-Encoding | gzip | | User-Agent | WWW-Mechanize/1.83 | '-----------------+--------------------' ==> 500 Can't connect to google.com:443 .-----------------+-------------------------------. | Response Header | Value | +-----------------+-------------------------------+ | Client-Date | Mon, 13 Feb 2017 18:04:23 GMT | | Client-Warning | Internal response | | Content-Type | text/plain | '-----------------+-------------------------------' .--------------------------------------------------------------------- +--------------------------------------------------------------------- +---. | Content + + | +--------------------------------------------------------------------- +--------------------------------------------------------------------- +---+ | Can't connect to google.com:443 + + | | + + | | Name or service not known at /usr/lib/perl5/site_perl/5.18.2/LWP/Pro +tocol/http.pm line 47, <DATA> line 1. + | '--------------------------------------------------------------------- +--------------------------------------------------------------------- +---' .--------------------------------------------------------------------- +--------------------------------------------------------------------- +---. | Text + + | +--------------------------------------------------------------------- +--------------------------------------------------------------------- +---+ | Can't connect to google.com:443 + + | | + + | | Name or service not known at /usr/lib/perl5/site_perl/5.18.2/LWP/Pro +tocol/http.pm line 47, <DATA> line 1. + | '--------------------------------------------------------------------- +--------------------------------------------------------------------- +---' Error GETing https://google.com/: Can't connect to google.com:443 at t +est.pl line 22.
      Thanks for taking the time. I have at least learned a bit more about the web tools available in Perl :)

      Cheers,
      R.

      Pereant, qui ante nos nostra dixerunt!

        Unfortunately, LWP::ConsoleLogger doesn't seem to be talkative about when it is trying to connect to a proxy.

        You can try to find which proxy LWP will try to use:

        #!perl -w use strict; use LWP::ConsoleLogger::Easy 'debug_ua'; use Data::Dumper; use LWP::UserAgent; my $ua=LWP::UserAgent->new(); warn Dumper $ua->proxy('https'); my $logger=debug_ua($ua,5); print Dumper \%ENV; $ua->env_proxy; warn Dumper $ua->proxy('https'); $ua->get('https://google.com')

        This should show you that your proxy URL is in %ENV and also picked up from there in the $ua.

        Maybe your proxy does not let Perl through? Have you tried masquerading your Perl script as curl (because that seems to work?)?:

        $ua->default_headers->header('User-Agent' => 'curl/7.10.x (i386-redhat +-linux-gnu) libcurl/7.10.x OpenSSL/0.9.7a ipv6 zlib/1.1.4');
Re: HTTPS returning 400 from Linux but 200 OK from Windows
by shadowsong (Monk) on Jan 27, 2017 at 15:09 UTC

    Hi R

    As Anonymous said above… what do you get when you try: lwp-request -m get -USsed  http://ppm.activestate.com/CPAN/authors  http://cpan.perl.org/authors?

    When I do it - I get:

    C:\Users\shadowsong>lwp-request -m get -USsed http://ppm.activestate. +com/CPAN/authors http://cpan.perl.org/authors GET http://ppm.activestate.com/CPAN/authors User-Agent: lwp-request/6.15 libwww-perl/6.15 407 authenticationrequired Cache-Control: no-cache Date: Fri, 27 Jan 2017 15:05:31 GMT Via: 1.1 10.156.69.200 (McAfee Web Gateway 7.6.2.21509) Proxy-Authenticate: NTLM Content-Length: 3895 Content-Type: text/html Client-Date: Fri, 27 Jan 2017 15:05:31 GMT Client-Peer: 10.156.69.200:3128 Client-Response-Num: 1 Client-Warning: Unsupported authentication scheme 'ntlm' Link: </mwg-internal/de5fs23hu73ds/files/default/stylesheet.css>; rel= +"stylesheet" Proxy-Connection: Keep-Alive Title: McAfee Web Gateway - Notification X-UA-Compatible: IE=7 GET http://cpan.perl.org/authors User-Agent: lwp-request/6.15 libwww-perl/6.15 407 authenticationrequired Cache-Control: no-cache Date: Fri, 27 Jan 2017 15:05:31 GMT Via: 1.1 10.156.69.200 (McAfee Web Gateway 7.6.2.21509) Proxy-Authenticate: NTLM Content-Length: 3873 Content-Type: text/html Client-Date: Fri, 27 Jan 2017 15:05:31 GMT Client-Peer: 10.156.69.200:3128 Client-Response-Num: 1 Client-Warning: Unsupported authentication scheme 'ntlm' Link: </mwg-internal/de5fs23hu73ds/files/default/stylesheet.css>; rel= +"stylesheet" Proxy-Connection: Keep-Alive Title: McAfee Web Gateway - Notification X-UA-Compatible: IE=7

    Shadowsong

      I get those CPAN urls OK. It looks like the issue is only with HTTPS sites. I have HTTP_PROXY and HTTPS_PROXY set including user/pass details in the form HTTP_PROXY='http://user:pass@proxy.my.net:8080'

      me@linuxbox:~> lwp-request -m get -USsed http://ppm.activestate.com/C +PAN/authors http://cpan.perl.org/authors? GET http://ppm.activestate.com/CPAN/authors/ User-Agent: lwp-request/5.810 GET http://ppm.activestate.com/CPAN/authors --> 301 Moved Permanently GET http://ppm.activestate.com/CPAN/authors/ --> 200 OK Cache-Control: proxy-revalidate Connection: close Date: Fri, 27 Jan 2017 16:11:05 GMT Age: 0 Server: Apache Content-Length: 4728 Content-Type: text/html;charset=UTF-8 Client-Date: Fri, 27 Jan 2017 16:11:53 GMT Client-Peer: 10.244.0.50:83 Client-Response-Num: 1 Title: Index of /CPAN/authors GET http://www.cpan.org/authors/? User-Agent: lwp-request/5.810 GET http://cpan.perl.org/authors? --> 301 Moved Permanently GET http://www.cpan.org/authors? --> 301 Moved Permanently GET http://www.cpan.org/authors/? --> 200 OK Cache-Control: proxy-revalidate Connection: close Date: Fri, 27 Jan 2017 16:11:54 GMT Via: 1.1 varnish-v4 Via: 1.1 varnish Via: 1.1 varnish Accept-Ranges: bytes Age: 0 Server: Apache/2.2.15 (CentOS) Vary: Accept-Encoding Content-Length: 4722 Content-Type: text/html;charset=UTF-8 Client-Date: Fri, 27 Jan 2017 16:11:54 GMT Client-Peer: 10.244.0.50:83 Client-Response-Num: 1 Fastly-Debug-Digest: 492dd663e1697be6e60aeecd19b5bf028765c1b3d66167d36 +df53fb8eefe26c0 Title: Index of /authors X-Cache: MISS, MISS X-Cache-Hits: 0, 0 X-Served-By: cache-lax8636-LAX, cache-bma7023-BMA X-Timer: S1485533514.441262,VS0,VE504
      Now if I go for https://google.com
      me@linuxbox:~> lwp-request -m get -USsed https://google.com GET https://google.com User-Agent: lwp-request/5.810 GET https://google.com --> 400 Bad Request Cache-Control: no-cache Connection: close Pragma: no-cache Content-Length: 677 Content-Type: text/html; charset=utf-8 Client-Date: Fri, 27 Jan 2017 16:16:23 GMT Client-Peer: 10.244.0.50:83 Client-Response-Num: 1 Proxy-Connection: close Title: Request Error
      At this site I have the same proxt settings for http and https, which strikes me as odd, but works fine on my windows lappy.

      Cheers,
      R.

      Pereant, qui ante nos nostra dixerunt!

        Hi R

        Hmmm, try setting the following environment variables:

        HTTP_PROXY_USER=your_proxy_username HTTP_PROXY_PASS=your_proxy_password

        And update HTTP_PROXY to:

        HTTP_PROXY=http://proxy.my.net:8080

        Best of luck!
        Shadowsong

Re: HTTPS returning 400 from Linux but 200 OK from Windows
by Anonymous Monk on Jan 27, 2017 at 09:00 UTC
Re: HTTPS returning 400 from Linux but 200 OK from Windows
by madtoperl (Pilgrim) on Jan 27, 2017 at 14:32 UTC
    You can try below,
    $ua->credentials( $linktologin, $pause, $username, $password ); Eg. $ua->credentials( 'https://google.com/:443', 'PAUSE', ‘$username +’, ‘$password’);

    You can refer below link for more details, https://perlmaven.com/lwp-useragent-and-basic-authentication

      https://google.com/:443 isn't a valid URL, even if it were Google wouldn't identify as 'PAUSE'. If you're going to copy and paste an example, it doesn't make sense to edit it so that it doesn't work.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1180446]
Approved by ww
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (4)
As of 2018-07-23 02:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    It has been suggested to rename Perl 6 in order to boost its marketing potential. Which name would you prefer?















    Results (459 votes). Check out past polls.

    Notices?