Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

LWP::Simple not getting content

by jonnyfolk (Vicar)
on Jul 26, 2018 at 12:32 UTC ( #1219309=perlquestion: print w/replies, xml ) Need Help??

jonnyfolk has asked for the wisdom of the Perl Monks concerning the following question:

I am using LWP::Simple to grab content from a page, and parse the data for exchange rates.

The url of the page is: https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml?025a000c2e476c719095c900b9c8b5e4

If I go to the page in my browser I can see the (xml) content. Using the script it is not getting the content. Why might this be? (It used to work!).

#!/usr/bin/perl -w use strict; use CGI::Carp qw(fatalsToBrowser warningsToBrowser); use CGI ':standard'; use LWP::Simple; my $address = 'https://www.ecb.europa.eu/stats/eurofxref/eurofxref-dai +ly.xml?025a000c2e476c719095c900b9c8b5e4'; my $content = get("$address"); die "Couldn't get it!" unless defined $content; print "Content-type:text/html\n\n"; print $content;

Replies are listed 'Best First'.
Re: LWP::Simple not getting content
by hippo (Chancellor) on Jul 26, 2018 at 12:40 UTC
    Why might this be?

    Could be almost anything, but my guess is on the certificate issuer not being in your CA list. The current cert was only issued on the 7th of June so that's a recent change. What version of Mozilla::CA are you using? It works fine with mine (20180117).

      Thanks for your reply. I switched over to LWP::UserAgent and successfully used the following code

      my $url = 'https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.x +ml?025a000c2e476c719095c900b9c8b5e4'; my $ua = LWP::UserAgent->new( ssl_opts => { SSL_verify_mode => 'SSL_VERIFY_NONE'}, ); $ua->agent('Mozilla/4.0 (compatible; MSIE 5.12; Mac_PowerPC)' ); my $response = $ua->get($url); if ($response->is_success) { print $response->decoded_content; # or whatever } else { die $response->status_line; }
Re: LWP::Simple not getting content
by Corion (Pope) on Jul 26, 2018 at 12:38 UTC

    Maybe the remote server does not like the LWP::Simple useragent header?

    My suggestion is to use something like LWP::ConsoleLogger and to compare and adapt what your script sends with what your browser sends until the two match, or you get a satisfactory result.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1219309]
Approved by Laurent_R
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (4)
As of 2021-09-27 13:26 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?