Beefy Boxes and Bandwidth Generously Provided by pair Networks chromatic writing perl on a camel
Keep It Simple, Stupid
 
PerlMonks  

LWP::UserAgent and Cookies

by tomazos (Deacon)
on Dec 12, 2001 at 01:32 UTC ( #131056=perlquestion: print w/ replies, xml ) Need Help??
tomazos has asked for the wisdom of the Perl Monks concerning the following question:

I want to get the content of a web page at a specific remote HTTP URL into a scalar.

I setup a script using LWP::UserAgent to do this, but it did not work because the server that the page is on requires the HTTP client to accept a cookie. It appears that LWP::UserAgent does not do this by default.

My code is as follows:

use strict; use warnings; use LWP::UserAgent; my $ua = new LWP::UserAgent; $ua->agent("Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC)"); my $req = new HTTP::Request GET => 'http://groups.yahoo.com/group/some +group/message/1243'; my $res = $ua->request($req); if ($res->is_success) { my $pagecontent = $res->content; } else { print "Bad luck this time\n"; }
Is it possible to tell LWP::UserAgent to accept a cookie? If not, is there another way of doing what I am trying to do?

Comment on LWP::UserAgent and Cookies
Download Code
Re: LWP::UserAgent and Cookies
by athomason (Curate) on Dec 12, 2001 at 02:06 UTC
    Sure, just set up a cookie jar using HTTP::Cookies:
    use strict; use LWP::UserAgent; use HTTP::Cookies; use HTTP::Request::Common; my $cookie_jar = HTTP::Cookies->new( ); my $ua = LWP::UserAgent->new; $ua->cookie_jar( $cookie_jar ); my $request = GET "http://www.doubleclick.net"; # don't worry, it does +n't get saved :-) my $response = $ua->request( $request ); $cookie_jar->extract_cookies( $response ); print $cookie_jar->as_string;
    Subsequent requests using the same user agent will send those cookies to the server. If you don't want them to be sent, use a new agent or clear the jar with $cookie_jar->clear.

    When you say the server requires that the client accept a cookie, presumably you mean that the cookie must be kept around in order to retrieve another page correctly. Because when serving the first page, the server has no idea whether the user will keep the cookie (i.e. send it back later). If you mean it must remember it and send it back, you'll want to reuse the user agent (and cookie jar) as mentioned above. If you want to preserve the cookie jar for later use outside your script, take a look at the save and load methods in HTTP::Cookies. For a longer discussion of this subject, check out perldoc lwpcook and poke through the Q&A section under the HTTP and FTP clients topic.

      Just a note:

      You don't need

      $cookie_jar->extract_cookies( $response );
      If cookie jar object is set for user agent object than user agent does this itself.

      --
      Ilya Martynov (http://martynov.org/)

(ichimunki) Re: LWP::UserAgent and Cookies
by ichimunki (Priest) on Dec 12, 2001 at 02:10 UTC
    You might want to check out 'perldoc lwpcook', short but sweet examples of doing a bunch of different things with LWP.
Re: LWP::UserAgent and Cookies
by Aighearach on Dec 12, 2001 at 02:15 UTC
    You need to use the $ua->cookie_jar method, which will require reading perldoc LWP::UserAgent and perldoc HTTP::Cookies.
    --
    Snazzy tagline here

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://131056]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (5)
As of 2014-04-20 11:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (485 votes), past polls