Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer

Parsing a page that requires login?

by Anonymous Monk
on Dec 29, 2000 at 19:19 UTC ( #48811=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I'm needing to parse through and display the HTML of for someone who's logged in. What I mean is, I am writing a script that will grab the source of, but each time it grabs the source the source will be different since each person's version of that page is different. What I want to know is, is this possible? Is it possible for me to create a login page, log someone in to and then grab the source off the page and display a parsed version of the html? And if so, how? Thanks, James

Replies are listed 'Best First'.
Re: Parsing a page that requires login?
by lhoward (Vicar) on Dec 29, 2000 at 19:40 UTC
    See my example here for how to use LWP with a site that does cookie style logins.
Re: Parsing a page that requires login?
by fongsaiyuk (Pilgrim) on Dec 29, 2000 at 20:02 UTC
    You should be able to accomplish this using the LWP module's methods. All you will need to know is the username and password for their page.

    If you are asking to check a person's site *without* knowing their username and password, well, see merlyn's post above and shame on you, you 7331 h4x0r! :)

    However, if I am understanding your question correctly, and you do have the username and password, you are able to create such "headless" clients using the LWP module. I know that there are some examples in the documentation

    Good Luck


Re: Parsing a page that requires login?
by merlyn (Sage) on Dec 29, 2000 at 19:22 UTC
      Could I be enlightened, anyway? Its for an HTML -> Palm Pilot converter; mainly I want it for my own use.
        use LWP :-)

        Have a nice day
        All decision is left to your taste
Re: Parsing a page that requires login?
by tekniko (Deacon) on Dec 29, 2000 at 19:41 UTC
    Heh, heh, you could build a database to store the login information of whomever you will be grabbing the content for, then write a program that will query the database each time for each user and grab the appropriate content for display. Not likely that you'll get anyone here to help you test this, however.
Re: Parsing a page that requires login?
by willdooUK (Beadle) on Apr 03, 2001 at 17:52 UTC
    Have you noticed the little hack that Yahoo use to stop their mail pages from cacheing?

    There's an id in the query string that's randomly generated every time you click on the mail button, so the URL is effectively new (and your browser won't recognise it and pull it from the local cache).

    Neat, if a little clunky.


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://48811]
Approved by root
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (11)
As of 2019-09-19 12:31 GMT
Find Nodes?
    Voting Booth?
    The room is dark, and your next move is ...

    Results (243 votes). Check out past polls.