Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Proper way to fetch html from a https site with PERL

by tollieman (Initiate)
on Nov 21, 2001 at 09:08 UTC ( #126726=perlquestion: print w/replies, xml ) Need Help??

tollieman has asked for the wisdom of the Perl Monks concerning the following question:

I would like to access a https: url with PERL to retreive the source from that site. What is the proper way to do this? I have used the following:
#!/usr/bin/perl use strict; use warnings; use LWP::Simple; checkit(); sub checkit { my $url = 'https://url.that.i.want'; my $content = get($url); my @data = split (/\n/, $content); foreach my $line (@data) { print "$line"; } }
I get the following error: 501 protocol scheme 'https' is not supported. along with the url that i requested printed to the screen. Any suggestions? Thanks

Replies are listed 'Best First'.
Re: Proper way to fetch html from a https site with PERL
by blakem (Monsignor) on Nov 21, 2001 at 09:48 UTC
    The encryption used in https isn't in the LWP modules... You'll need to install something like Net::SSL or Crypt::SSLeay which can work with the LWP modules to help you fetch https pages.

    -Blake

Re: Proper way to fetch html from a https site with PERL
by mkmcconn (Chaplain) on Nov 21, 2001 at 09:46 UTC
Re: Proper way to fetch html from a https site with PERL
by dws (Chancellor) on Nov 21, 2001 at 10:55 UTC
    If you are using ActiveState Perl,   C:\> ppm install Crypt-SSLeay Will install what you need to add https: support to LWP. It describes itself as:  Crypt-SSLeay [0.17.1] OpenSSL and SSLeay glue that provides LWP https support Works for me.

Re: Proper way to fetch html from a https site with PERL
by kwoff (Friar) on Nov 22, 2001 at 03:16 UTC
    The following isn't necessarily the "proper way", but it's something that came to mind. I find it useful when debugging Perl CGI stuff for https:// servers to use `stunnel`. It's like telnet but uses SSL protocol. Example:
    $ stunnel -cr foo.com:443 GET /cgi-bin/store.pl HTTP/1.0 HTTP/1.1 302 Moved Date: Wed, 21 Nov 2001 21:32:06 GMT Server: Apache/1.3.22 (Unix) mod_ssl/2.8.5 OpenSSL/0.9.6b location: /store/index.html Connection: close Content-Type: text/plain

    There I used it like a telnet client. You can also pipe stuff to it (actually, you can pipe stuff to telnet, too). I'm not sure offhand how to get LWP::Simple to use stunnel, though.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://126726]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (5)
As of 2022-06-28 22:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My most frequent journeys are powered by:









    Results (92 votes). Check out past polls.

    Notices?