Contributed by Anonymous Monk
on Apr 20, 2001 at 02:04 UTC
Q&A
> HTTP and FTP clients
Description: I want to GET data from a website. The problem is I need
to enter my username and password to login in on one page.
Then I can go to other pages to access data.
Using perl (from a command prompt), I can GET the HTML, but
the server thinks I'm a 'guest' so I get incomplete data.
How can I log in to one url and then GET data from another
url within the same server? Answer: How can I login to get data from a webpage? contributed by tinman This is part of a script that I wrote for getting data from a password protected site. The site uses cookies for the logins, and the password URL can be passed straight in via the URL as GET parameters.
use LWP::UserAgent;
use HTTP::Cookies;
$ua = LWP::UserAgent->new;
$ua->cookie_jar(HTTP::Cookies->new(file => "$cookie_file",
autosave => 1));
$ua->timeout($url_timeout);
$res = $ua->request(HTTP::Request->new('GET', "$login_url"));
if ($res->is_error()) {
$response = $response->status_line;
exit;
}
$response = $ua->request(HTTP::Request->new('GET', "$data_url"));
if ($response->is_error()) {
$response = $response->status_line;
}
else {
# do what you want to do with the pages here
}
First I create a "cookie jar" which retains the cookie submitted by the site login mechanism..
Then, I submit the login_url, which is a complete URL escaped string.
Thereafter, if the login is successful, I can navigate to any of the protected URLs..
HTH | Answer: How can I login to get data from a webpage? contributed by Daddio In addition to the code sample from tinman, check out CPAN's LWP, LWP::UserAgent, LWP::AuthenAgent.
Also, there are other nodes which may be of help here on PerlMonks that can help, too: 1, 2, or the general UserAgent search results.
These may give you a few pointers and more examples of ways to access the sites, whether or not the site is using cookies for login. |
Please (register and) log in if you wish to add an answer
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
Outside of code tags, you may need to use entities for some characters:
| |
For: |
|
Use: |
| & | | & |
| < | | < |
| > | | > |
| [ | | [ |
| ] | | ] |
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.
|
|