Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

LWP::UserAgent with HTTP::Cookies problems

by Alives (Initiate)
on Nov 08, 2005 at 22:18 UTC ( [id://506921]=perlquestion: print w/replies, xml ) Need Help??

Alives has asked for the wisdom of the Perl Monks concerning the following question:

Oh beloved, worshipped, sacred, holy Monks: I have written a script to grab a www page. I have to pass cookies in order to get the page. I can successfully get it, but it seems the cookies arent being used properly. I have asked around among all the perl people I know, but no one can answer this for me.
Can anyone find any problems in this code? The cookies file is correct and standardized... looks like this:
#LWP-Cookies-1.0 Set-Cookie3: id=bob_loblaw; path="/"; domain=yoursite.here.com; path_s +pec; expires="2005-12-31 23:59:33Z"; version=0 Set-Cookie3: username=blahblahblah; path="/"; domain=yoursite.here.com +; path_spec; expires="2015-11-06 08:09:49Z"; version=0
--------------------------
use strict; use warnings; use LWP::UserAgent; use HTTP::Cookies; my $rssFile = "dump.rss"; my $wwwSite = "www.your.site.here.com"; my $cookieFile = "c:/cookies.txt"; my $browser = LWP::UserAgent->new; $browser->timeout(10); $browser->env_proxy; $browser->cookie_jar(HTTP::Cookies->new(file => $cookieFile )); my $response = $browser->get( $wwwSite ); if ($response->is_success) { open RSSDUMP, ">" . $rssFile; print RSSDUMP $response->content; close RSSDUMP; } else { die $response->status_line; }

Replies are listed 'Best First'.
Re: LWP::UserAgent with HTTP::Cookies problems
by planetscape (Chancellor) on Nov 09, 2005 at 00:41 UTC
Re: LWP::UserAgent with HTTP::Cookies problems
by blahblahblah (Priest) on Nov 09, 2005 at 00:54 UTC
    Cookies are a pain to debug. It's helpful to see exactly what headers are being passed back and forth. There's a very cool tool called Charles that can help. It's a personal proxy server made for debugging things like this.

    Tell your perl script to use it and record that request/response. Then tell your browser to use it and record that. Then compare the two. Chances are some difference between the headers or the request content will jump out at you.

    As a separate note, are you sure the domain in your cookies is valid? I think cookie domains are supposed to start with a ".", like ".here.com". I'm not sure though. (That's something you could find out by proxying your browser.)

    -Joe

      Thanks for pointing out Charles - that may be useful in future. :-) Ooh, I shoulda mentioned Ethereal...

      planetscape
      thanks for the help. i ended up using mechanize. i really appreciate all this!!!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://506921]
Approved by friedo
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others goofing around in the Monastery: (4)
As of 2024-03-19 07:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found