Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

Downloading files from a Secured Site

by mantra2006 (Hermit)
on May 24, 2007 at 17:25 UTC ( #617277=perlquestion: print w/replies, xml ) Need Help??

mantra2006 has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks

I have to work on a program which will connect to a secured site give username&password and then download certain files.
I need some help on how to go about this using perl like what lib's are needed etc.

Its basically "https" site. The client provided us certificate which I installed in browser and able to connect
to their site using username and password. Now I have to automate this task using a program thats where I stopped.

Thanks & Regards

"Coming together is a beginning. Keeping together is progress. Working together is success.- Henry Ford

Replies are listed 'Best First'.
Re: Downloading files from a Secured Site
by imp (Priest) on May 24, 2007 at 17:32 UTC
    I like WWW::Mechanize.

    Something like this:

    use strict; use warnings; use WWW::Mechanize; my $mech = WWW::Mechanize->new; $mech->credentials('user', 'pass'); my $response = $mech->get(''); die $response->status_line unless $response->is_success; printf "content-type: %s\n", $response->header('Content-Type'); my $data = $response->content;
      In response to your update:
      You will need Crypt::SSLeay for https traffic. See the section on PKCS12 support.

      Updated example:

      use strict; use warnings; use WWW::Mechanize; use Crypt::SSLeay; $ENV{HTTPS_PKCS12_FILE} = 'certs/pkcs12.pkcs12'; $ENV{HTTPS_PKCS12_PASSWORD} = 'PKCS12_PASSWORD'; my $mech = WWW::Mechanize->new; $mech->credentials('user', 'pass'); my $response = $mech->get(''); die $response->status_line unless $response->is_success; printf "content-type: %s\n", $response->header('Content-Type'); my $data = $response->content;
Re: Downloading files from a Secured Site
by naikonta (Curate) on May 24, 2007 at 17:48 UTC
    The LWP::UserAgent allows you to create program to interact with the web server much like browsers do. WWW::Mechanize provides more advanced way to automate interaction process with web services. It's built on top of many modules and LWP::UserAgent is one of them.

    You might also like to use the search box on the top page or Super Search to find previous discussions about this, such as Secure Site Login with Mechanize.

    Open source softwares? Share and enjoy. Make profit from them if you can. Yet, share and enjoy!

Re: Downloading files from a Secured Site
by cengineer (Pilgrim) on May 24, 2007 at 17:29 UTC
    Can you give some more detail? How are you connecting to the site? i.e. what protocol are you planning on using? What do you mean by "secured"? Do you mean the connection is going to be encrypted?
Re: Downloading files from a Secured Site
by injunjoel (Priest) on May 24, 2007 at 17:42 UTC
    Well depending on what you are going to use to connect, there are a few options.
    And many more, but these are a good start.
    Happy Reading....
    Based on the Update I would suggest, as other have, using WWW::Mechanize

    "I do not feel obliged to believe that the same God who endowed us with sense, reason and intellect has intended us to forego their use." -Galileo
Re: Downloading files from a Secured Site
by zentara (Archbishop) on May 25, 2007 at 17:09 UTC
    UPDATE: Fixed code to handle auth, my previous script was untested :-( Here is a simple way with progress( tested :-) ):
    #!/usr/bin/perl -w #adapted from lwpcook use strict; use LWP::UserAgent; my $user = 'zentara'; my $pass = 'foobar'; my $ua = LWP::UserAgent->new; my $URL = ''; my $filename = substr( $URL, rindex( $URL, "/" ) + 1 ); #print "$filename\n"; open( IN, ">$filename" ) or die $!; print "Fetching $URL\n"; my $expected_length; my $bytes_received = 0; my $req = HTTP::Request->new(GET => $URL); $req->authorization_basic($user, $pass); my $res = $ua->request($req, sub { my ( $chunk, $res ) = @_; $bytes_received += length($chunk); unless ( defined $expected_length ) { $expected_length = $res->content_length || 0; } if ($expected_length) { printf STDERR "%d%% - ", 100 * $bytes_received / $expected +_length; } print STDERR "$bytes_received bytes received\n"; # XXX Should really do something with the chunk itself print IN $chunk; } ); print $res->status_line, "\n"; close IN; exit;

    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
      Thank you, very interesting piece :) It even owes a post of its own in the Code Catacombs!

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://617277]
Approved by naikonta
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (7)
As of 2021-03-02 15:00 GMT
Find Nodes?
    Voting Booth?
    My favorite kind of desktop background is:

    Results (51 votes). Check out past polls.