Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

LWP::UserAgent with HTTP::Cookies problems

by Alives (Initiate)
on Nov 08, 2005 at 22:18 UTC ( #506921=perlquestion: print w/ replies, xml ) Need Help??
Alives has asked for the wisdom of the Perl Monks concerning the following question:

Oh beloved, worshipped, sacred, holy Monks: I have written a script to grab a www page. I have to pass cookies in order to get the page. I can successfully get it, but it seems the cookies arent being used properly. I have asked around among all the perl people I know, but no one can answer this for me.
Can anyone find any problems in this code? The cookies file is correct and standardized... looks like this:
#LWP-Cookies-1.0 Set-Cookie3: id=bob_loblaw; path="/"; domain=yoursite.here.com; path_s +pec; expires="2005-12-31 23:59:33Z"; version=0 Set-Cookie3: username=blahblahblah; path="/"; domain=yoursite.here.com +; path_spec; expires="2015-11-06 08:09:49Z"; version=0
--------------------------
use strict; use warnings; use LWP::UserAgent; use HTTP::Cookies; my $rssFile = "dump.rss"; my $wwwSite = "www.your.site.here.com"; my $cookieFile = "c:/cookies.txt"; my $browser = LWP::UserAgent->new; $browser->timeout(10); $browser->env_proxy; $browser->cookie_jar(HTTP::Cookies->new(file => $cookieFile )); my $response = $browser->get( $wwwSite ); if ($response->is_success) { open RSSDUMP, ">" . $rssFile; print RSSDUMP $response->content; close RSSDUMP; } else { die $response->status_line; }

Comment on LWP::UserAgent with HTTP::Cookies problems
Select or Download Code
Re: LWP::UserAgent with HTTP::Cookies problems
by planetscape (Canon) on Nov 09, 2005 at 00:41 UTC
Re: LWP::UserAgent with HTTP::Cookies problems
by blahblahblah (Priest) on Nov 09, 2005 at 00:54 UTC
    Cookies are a pain to debug. It's helpful to see exactly what headers are being passed back and forth. There's a very cool tool called Charles that can help. It's a personal proxy server made for debugging things like this.

    Tell your perl script to use it and record that request/response. Then tell your browser to use it and record that. Then compare the two. Chances are some difference between the headers or the request content will jump out at you.

    As a separate note, are you sure the domain in your cookies is valid? I think cookie domains are supposed to start with a ".", like ".here.com". I'm not sure though. (That's something you could find out by proxying your browser.)

    -Joe

      thanks for the help. i ended up using mechanize. i really appreciate all this!!!

      Thanks for pointing out Charles - that may be useful in future. :-) Ooh, I shoulda mentioned Ethereal...

      planetscape

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://506921]
Approved by friedo
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (5)
As of 2014-09-17 07:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (65 votes), past polls