How can I login to get data from a webpage?

[Anonymous Monk]

I want to GET data from a website. The problem is I need to enter my username and password to login in on one page. Then I can go to other pages to access data. Using perl (from a command prompt), I can GET the HTML, but the server thinks I'm a 'guest' so I get incomplete data. How can I log in to one url and then GET data from another url within the same server?

Originally posted as a Categorized Question.

[tinman]

This is part of a script that I wrote for getting data from a password protected site. The site uses cookies for the logins, and the password URL can be passed straight in via the URL as GET parameters.

use LWP::UserAgent;
use HTTP::Cookies;

$ua = LWP::UserAgent->new;
$ua->cookie_jar(HTTP::Cookies->new(file => "$cookie_file",
                                     autosave => 1));
$ua->timeout($url_timeout);
$res = $ua->request(HTTP::Request->new('GET', "$login_url"));

if ($res->is_error()) {
     $response = $response->status_line;
     exit;
 }
 
$response = $ua->request(HTTP::Request->new('GET', "$data_url")); 
if ($response->is_error()) {
     $response = $response->status_line;
 }
 else {
   # do what you want to do with the pages here
}

First I create a "cookie jar" which retains the cookie submitted by the site login mechanism.. Then, I submit the login_url, which is a complete URL escaped string. Thereafter, if the login is successful, I can navigate to any of the protected URLs..
HTH

[Daddio]

In addition to the code sample from tinman, check out CPAN's LWP, LWP::UserAgent, LWP::AuthenAgent.

Also, there are other nodes which may be of help here on PerlMonks that can help, too: 1, 2, or the general UserAgent search results.

These may give you a few pointers and more examples of ways to access the sites, whether or not the site is using cookies for login.