Get url

kepler has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Get url by huck (Prior) on Apr 29, 2017 at 21:45 UTC
And farther on #!/usr/bin/perl use strict; use warnings; use LWP; my $url = "https://www.earthquake.usgs.gov/earthquakes/feed/v1.0/sum +mary/2.5_week.atom"; my $ua; $ua = new LWP::UserAgent(keep_alive=>1 ,agent =>"Mozilla/5.0 (Windows NT 5.1; rv:51 +.0) Gecko/20100101 Firefox/51.0" ,ssl_opts => { verify_hostname => 0 , SSL_verify_mode => 'SSL_VERIF +Y_NONE' } # ,timeout=>$timeout ); my $req = new HTTP::Request (GET => $url); my $response = $ua->request ($req); unless ($response->is_success) { die $response->status_line; } [download] result `404 Not Found at 1189178a.pl line 20.` [download]	[reply] [d/l] [select]
Re^2: Get url by pryrt (Abbot) on Apr 29, 2017 at 22:31 UTC
with my browser (on my phone right now, so no easy perl), I verified that while https gives 404, http gives atom file. kepler should try your error-checking with correct protocol.	[reply]
Re: Get url by Discipulus (Canon) on Apr 29, 2017 at 21:33 UTC
hello kepler at first glance and using LWP::UserAgent perl -MLWP::UserAgent -e "print LWP::UserAgent->new->get($ARGV[0])->co +ntent" https://www.earthquake.usgs.gov/earthquakes/feed/v1.0/summary/ +2.5_week.atom Can't connect to www.earthquake.usgs.gov:443 (certificate verify faile +d) LWP::Protocol::https::Socket: SSL connect attempt failed with unknown +errorerror:14090086:SSL routines:SSL3_GET_SERVER_C ERTIFICATE:certificate verify failed at C:/ulisse/strawberry/perl/vend +or/lib/LWP/Protocol/http.pm line 51. #another tool same response: Response code: 500 Response message: Can't connect to www.earthquake.usgs.gov:443 + (certificate verify failed) [download] But telling LWP::UserAgent to not check ssl `LWP::UserAgent->new(ssl_opts => { verify_hostname => 0 })` all is fine: `perl -MLWP::UserAgent -e "print LWP::UserAgent->new(ssl_opts => { veri +fy_hostname => 0 )->get($ARGV[0])->content" https://www.earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week +.atom <!DOCTYPE HTML> <html> <head> <meta charset="utf-8"> <title>USGS URL Resolution Error Page</title> [...]` [download] L* There are no rules, there are no thumbs.. Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.	[reply] [d/l] [select]
Re: Get url by shmem (Chancellor) on Apr 29, 2017 at 21:39 UTC
Retrieving this url with firefox gives me: Your connection is not secure The owner of www.earthquake.usgs.gov has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website. Learn more… So it looks like LWP::Simple does https requests with certificate validation turned on by default. update: So use something like this: `perl -MLWP::UserAgent -e 'print LWP::UserAgent->new(ssl_opts => {verif +y_hostname => 0})->get($ARGV[0])->content' https://www.earthquake.usg +s.gov/earthquakes/feed/v1.0/summary/2.5_week.atom` [download] perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'	[reply] [d/l]
Re: Get url by NetWallah (Canon) on Apr 30, 2017 at 01:01 UTC
This url returns a 404 : https://www.earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.atom If you drop the "www.", it works fine: https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.atom ...Disinformation is not as good as datinformation. Don't document the program; program the document.	[reply]


Perl Monk, Perl Meditation
	PerlMonks