Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

LWP::Simple::get($url) does not work for particular urls

by ssara (Acolyte)
on Dec 31, 2017 at 13:57 UTC ( #1206484=perlquestion: print w/replies, xml ) Need Help??
ssara has asked for the wisdom of the Perl Monks concerning the following question:

I am using LWP::Simple::get($url) library to access data from web pages. The problem is that the get function is not working for the below url. Below is the code snippet:

#!/usr/bin/perl use LWP::Simple; use JSON; use Data::Dumper; my $url = "https://www.cryptopia.co.nz/api/GetCurrencies"; my $json = get( $url); die "Could not get $url!" unless defined $json; my $decoded_json = decode_json($json); print Dumper($decoded_json);

After running this code it gives the below error: Could not get https://www.cryptopia.co.nz/api/GetCurrencies! When i replace the url with :

 $url = "https://api.coinmarketcap.com/v1/ticker/"

it works fine. Please can you tell me what is the root cause and how I can fix it. Also the url mentioned in the code snippet worked once and now suddenly it does not work.

Replies are listed 'Best First'.
Re: LWP::Simple::get($url) does not work for particular urls (SSL certificate verification)
by 1nickt (Monsignor) on Dec 31, 2017 at 14:48 UTC

    Hi, using getprint() instead shows the error:

    500 Can't connect to www.cryptopia.co.nz:443 (certificate verify faile +d) <URL:https://www.cryptopia.co.nz/api/GetCurrencies>
    ( ... because, as the doc for LWP::Simple states:
    "The get() function will fetch the document identified by the given URL and return it. It returns undef if it fails."
    While for getprint():
    "If the request fails, then the status code and message are printed on STDERR. The return value is the HTTP response code."
    )

    You should use the full LWP::UserAgent so you can disable SSL certificate checking, or use HTTP::Tiny, which has it disabled by default:

    use strict; use warnings; use HTTP::Tiny; use JSON; use Data::Dumper; my $url = "https://www.cryptopia.co.nz/api/GetCurrencies"; my $res = HTTP::Tiny->new->get( $url ); my $decoded_json = decode_json( $res->{'content'} ); print Dumper( $decoded_json ); __END__
    Partial output:
    $ perl 1206484.pl | head -20 $VAR1 = { 'Message' => undef, 'Error' => undef, 'Data' => [ { 'Status' => 'OK', 'StatusMessage' => undef, 'DepositConfirmations' => 20, 'Algorithm' => 'POS', 'Name' => '1337', 'MaxWithdraw' => '90000000', 'MinTip' => '166.66666666', 'Symbol' => '1337', 'Id' => 331, 'ListingStatus' => 'Active', 'IsTipEnabled' => bless( do{\(my $o = 0)}, 'JS +ON::PP::Boolean' ), 'WithdrawFee' => '0.01', 'MinWithdraw' => '20000', 'MinBaseTrade' => '2e-05' },

    Hope this helps!


    The way forward always starts with a minimal test.
      >You should use the full LWP::UserAgent so you can disable SSL certificate checking, or use HTTP::Tiny, which has it disabled by default:

      Certificate validation is there for a reason and simply recommending to switch it off is essentially suggesting to abandon any security provided by https since man in the middle attacks are easy if proper certificate validation is not done. The real reason that the code fails is a broken setup of the target site. This can be worked around in a secure way by setting SSL_ca_file to the appropriate CA certificates or by using SSL_fingerprint. For details see Stackoverflow where the question was also asked and I've answered it in detail.

      Thanks a lot.I have one query regarding valid certificate.If there was issue of SSL certificate then why i am directly able to see the data in web browser when i browse the link https://www.cryptopia.co.nz/api/GetCurrencies in the web browser?

        If there was issue of SSL certificate then why i am directly able to see the data in web browser when i browse the link https://www.cryptopia.co.nz/api/GetCurrencies in the web browser?

        Hi, this is because your desktop browser uses the Certificate Authority root certificates that are distributed with your operating system, whereas LWP::UserAgent does not know about them. You can force LWP to use the same root certificates by setting the value of the environment variable HTTPS_CA_DIR to the location on your system.

        On my system the certificate directory is /etc/ssl/certs.

        Without:

        use strict; use warnings; use LWP::UserAgent; use JSON; use Data::Dumper; my $url = "https://www.cryptopia.co.nz/api/GetCurrencies"; my $res = LWP::UserAgent->new->get( $url ); die $res->content if not $res->is_success; print Dumper decode_json $res->decoded_content; __END__
        Output:
        $ perl 1206484.pl | head -20 Can't connect to www.cryptopia.co.nz:443 (certificate verify failed) SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server +_certificate:certificate verify failed at /home/nick/perl5/perlbrew/p +erls/perl-5.26.1/lib/site_perl/5.26.1/LWP/Protocol/http.pm line 46.
        Now with the UA pointed at the CA cert store:
        use strict; use warnings; use LWP::UserAgent; use JSON; use Data::Dumper; BEGIN { $ENV{'HTTPS_CA_DIR'} = '/etc/ssl/certs'; } my $url = "https://www.cryptopia.co.nz/api/GetCurrencies"; my $res = LWP::UserAgent->new->get( $url ); die $res->content if not $res->is_success; print Dumper decode_json $res->decoded_content; __END__
        Output:
        $ perl 1206484.pl | head -20 $VAR1 = { 'Message' => undef, 'Success' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ), 'Error' => undef, 'Data' => [ { 'MaxWithdraw' => '90000000', 'IsTipEnabled' => bless( do{\(my $o = 0)}, 'JS +ON::PP::Boolean' ), 'DepositConfirmations' => 20, 'Status' => 'OK', 'MinWithdraw' => '20000', 'Name' => '1337', 'Algorithm' => 'POS', 'Symbol' => '1337', 'MinBaseTrade' => '2e-05', 'StatusMessage' => undef, 'MinTip' => '166.66666666', 'ListingStatus' => 'Active', 'Id' => 331, 'WithdrawFee' => '0.01'

        Hope this helps!


        The way forward always starts with a minimal test.
        Maybe your browser uses different certificates than your program?

        ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord }map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,
Re: LWP::Simple::get($url) does not work for particular urls
by Marshall (Abbot) on Jan 02, 2018 at 04:37 UTC
    I ran your code on my WinXP laptop and it works just fine.

    I suspect that you need to install: Crypt-SSLeay.

    Update: I didn't do anything special. This is an Active State std distribution with Crypt-SSLeay installed by me manually.

    >perl -v This is perl 5, version 20, subversion 2 (v5.20.2) built for MSWin32-x +86-multi-thread-64int (with 1 registered patch, see perl -V for more detail) Copyright 1987-2015, Larry Wall Binary build 2002 [299195] provided by ActiveState http://www.ActiveSt +ate.com Built Jul 20 2015 13:29:53
    Further comment/question:
    I commend the OP for posting clear, short code that I was able to run! If the post was lengthy or complicated, I probably wouldn't have bothered. But I could do this easily with some cut-n-paste. I don't get any errors running the OP's code verbatim.

    The OP's question was "Why doesn't this work?". My question is "Why does this work?". The only thing I could think of was that I do have this Crypt module installed. I am not sure what is going on "under the covers" that allows my machine to successfully access my $url = "https://www.cryptopia.co.nz/api/GetCurrencies";. My experience with https sites is limited and I profess no expertise whatsoever, but I've never seen an https access fail as long as I have this Crypt module installed.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1206484]
Approved by davies
Front-paged by haukex
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (5)
As of 2018-06-24 08:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?



    Results (126 votes). Check out past polls.

    Notices?