Downloading files from a Secured Site

[mantra2006]

Hello Monks

I have to work on a program which will connect to a secured site give username&password and then download certain files.
I need some help on how to go about this using perl like what lib's are needed etc.

Update:
Its basically "https" site. The client provided us certificate which I installed in browser and able to connect
to their site using username and password. Now I have to automate this task using a program thats where I stopped.

[imp]

I like WWW::Mechanize.

Something like this:

use strict;
use warnings;
use WWW::Mechanize;
my $mech = WWW::Mechanize->new;
$mech->credentials('user', 'pass');
my $response = $mech->get('http://foo.com');

die $response->status_line unless $response->is_success;

printf "content-type: %s\n", $response->header('Content-Type');

my $data = $response->content;

[imp]

In response to your update:
You will need Crypt::SSLeay for https traffic. See the section on PKCS12 support.

Updated example:

use strict;
use warnings;
use WWW::Mechanize;
use Crypt::SSLeay;

$ENV{HTTPS_PKCS12_FILE}     = 'certs/pkcs12.pkcs12';
$ENV{HTTPS_PKCS12_PASSWORD} = 'PKCS12_PASSWORD';

my $mech = WWW::Mechanize->new;
$mech->credentials('user', 'pass');
my $response = $mech->get('https://foo.com');

die $response->status_line unless $response->is_success;

printf "content-type: %s\n", $response->header('Content-Type');

my $data = $response->content;

[naikonta]

The LWP::UserAgent allows you to create program to interact with the web server much like browsers do. WWW::Mechanize provides more advanced way to automate interaction process with web services. It's built on top of many modules and LWP::UserAgent is one of them.

You might also like to use the search box on the top page or Super Search to find previous discussions about this, such as Secure Site Login with Mechanize.

[cengineer]

Can you give some more detail? How are you connecting to the site? i.e. what protocol are you planning on using? What do you mean by "secured"? Do you mean the connection is going to be encrypted?

[injunjoel]

Well depending on what you are going to use to connect, there are a few options.

• Net::SCP
• Net::SFTP
And many more, but these are a good start.
Happy Reading....
Update:
Based on the Update I would suggest, as other have, using WWW::Mechanize

-InjunJoel

[zentara]

UPDATE: Fixed code to handle auth, my previous script was untested :-( Here is a simple way with progress( tested :-) ):

#!/usr/bin/perl -w
#adapted from lwpcook
use strict;
use LWP::UserAgent;

my $user = 'zentara';
my $pass = 'foobar';

my $ua = LWP::UserAgent->new;

my $URL = 'https://zentara.zentara.net/~zentara/zentara1.avi';
my $filename = substr( $URL, rindex( $URL, "/" ) + 1 );
#print "$filename\n";
open( IN, ">$filename" ) or die $!;

print "Fetching $URL\n";

my $expected_length;
my $bytes_received = 0;

my $req = HTTP::Request->new(GET => $URL);
$req->authorization_basic($user, $pass);

my $res = $ua->request($req,
    sub {
        my ( $chunk, $res ) = @_;
        $bytes_received += length($chunk);
        unless ( defined $expected_length ) {
            $expected_length = $res->content_length || 0;
        }
        if ($expected_length) {
            printf STDERR "%d%% - ", 100 * $bytes_received / $expected
+_length;
        }
        print STDERR "$bytes_received bytes received\n";

        # XXX Should really do something with the chunk itself
        print IN $chunk;
    }
);

print $res->status_line, "\n";
close IN;
exit;

[wazoox]

Thank you, very interesting piece :) It even owes a post of its own in the Code Catacombs!