Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

I am trying to access data from a site that needs login and password

by aykes (Initiate)
on Apr 20, 2001 at 10:59 UTC ( #74107=perlquestion: print w/replies, xml ) Need Help??

aykes has asked for the wisdom of the Perl Monks concerning the following question:

I am very new to Perl and really need some help. The webpage where I need to enter my login and password is: http://baseball.fantasysports.yahoo.com/baseball The page I want to view after submitting the form is: http://baseball.fantasysports.yahoo.com/baseball/show?page=roster&lid=23413&mid=2 How can I submit the form, then view the second webpage source code at a command prompt? Can someone give me code for this, or point me in the right direction? I've tried some of the suggestions that other have written, but have been unsuccessful. Is it possible to do with this website?

Replies are listed 'Best First'.
Re: I am trying to access data from a site that needs login and password
by tinman (Curate) on Apr 20, 2001 at 11:09 UTC

    Try the code in Answer: How can I login to get data from a webpage? as a starting point...

    I took a look at the source code for the page that you gave... in your case, the "GET" method should be replaced with the "POST" method... and the login URL is http://login.yahoo.com/config/login

    What you need to do is to construct the values for the variables within the login form (ie: login, password) and POST it into the form URL... Thereafter, if the login is successful, a simple web page fetch should work for you..
    HTH

      I tried using this code. It saved the cookies to a file, but I don't know how to use them or if I have to use them. But I think I am posting properly, but at the end of the code where I GET the data, it just gives me 'guest' data as if I didn't log in.
      use HTTP::Response; use HTTP::Request; use LWP::UserAgent; use HTTP::Request::Common; use HTTP::Cookies; $login = "username"; $password = "password"; $action = "http://login.yahoo.com/config/login"; $cookie_file = "d:/perl/yahoo_cookie.txt"; $cookie_jar = HTTP::Cookies->new(file=>"$cookie_file"); $ua = LWP::UserAgent->new(); $ua->agent("Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)"); $ua->timeout(600); $ua->cookie_jar(HTTP::Cookies->new(file => "$cookie_file", autosave => + 1)); $response= $ua->request(POST $action,[ '.tries'=>"1", '.done'=>"http://baseball.fantasysports.yahoo.com/baseball", '.src'=>"", 'lg'=>"us", '.intl'=>"us", 'login'=>$login, 'passwd'=>$password, '.chkp'=>"Y"]); $cookie_jar->extract_cookies($response); $content = $response->content(); print $response->as_string; $team1 = "http://baseball.fantasysports.yahoo.com/baseball/show?pag +e=roster&lid=23413&mid=2"; use LWP::Simple; $content2 = get($team1); print $content2;
      Is there something wrong with this? Am I GETing the data of the next page properly? By the way I am trying to do all of this in a DOS window. Can I do it that way? By the questions I am asking, I'm sure you can tell that I am new to this. I hope you can help. Thanks.
Re: I am trying to access data from a site that needs login and password
by plazm (Initiate) on Apr 20, 2001 at 18:33 UTC
    On your pages use CGI.pm to parse out CGI input. Do perldoc on CGI.pm.

    To view the output of the second page you have simulate form input. To do this, start the second page script from command line and then give it name=value pairs. ^D end the input. You will get your output back along with HTTP headers.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://74107]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (8)
As of 2020-07-13 10:06 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?