Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Simulating --keep-session-cookies wget option with LWP ?

by Corion (Pope)
on Sep 27, 2011 at 09:36 UTC ( #928054=note: print w/ replies, xml ) Need Help??


in reply to Simulating --keep-session-cookies wget option with LWP ?

As you don't show the code using LWP::UserAgent, it's hard for me to guess what you're doing wrong (besides that).

My guess is that you are directly overwriting the cookie file in your unshown program. I use WWW::Mechanize (not WWW::Mechandize) with (persistent) cookies and it works well. As WWW::Mechanize just uses LWP::UserAgent, both should work well with cookies.

I doubt that the cookies file that wget writes is supposed to work with LWP::UserAgent resp. HTTP::Cookies, but maybe they use the same format. I haven't seen this documented anywhere.


Comment on Re: Simulating --keep-session-cookies wget option with LWP ?
Select or Download Code
Re^2: Simulating --keep-session-cookies wget option with LWP ?
by i5513 (Monk) on Sep 27, 2011 at 09:50 UTC

    I can read in Re: Passing a cookie with LWP::UserAgent about session cookies are deleted when browser (UserAgent in LWP) is closed, so every thread which I start will start with an new cookie

    My last try with HTTP::Cookies was:

    # try to simulate wget: my $cookie_jar = HTTP::Cookies->new( file => "$cookie", autosave => 1, ); my $ua = LWP::UserAgent->new ; $ua->timeout(2); $ua->cookie_jar($cookie_jar); $ua->credentials ("$host:$port","Tomcat Manager Application",$user,$pa +ssword); my $request= HTTP::Request->new (GET=> "http://$host:$port/manager/sta +tus?XML=true"); my $status=$ua->request ($request); if ($status->is_success) { $page=$status->decoded_content(); } else { print STDERR $status->status_line, "\n"; return 1; }
    Thanks!

      Your code works for me, once I fix it to actually become a program:

      use strict; use HTTP::Cookies; use LWP::UserAgent; # try to simulate wget: my $cookie_jar = HTTP::Cookies->new( file => "mycookie.cookie", autosave => 1, ); my $ua = LWP::UserAgent->new; $ua->timeout(2); $ua->cookie_jar($cookie_jar); $ua->env_proxy; if( !@ARGV) { warn "Requesting page"; my $request= HTTP::Request->new (GET=> "http://www.google.com/webhp?hl +=en"); my $status=$ua->request ($request); if ($status->is_success) { print $status->decoded_content(); } else { print STDERR $status->status_line, "\n"; return 1; } }; use Data::Dumper; warn "Cookies in jar:"; $cookie_jar->scan(sub { warn Dumper \@_ });

      It (re)generates the cookie file if run without any command line argument, and if run with a command line argument, it just dumps the cookies from the file.

      Whatever your problem seems to be, I would suspect that it lies elsewhere.

        Ok, I tested your script with google and it works.

        Then it maybe related with tomcat auth issue:
        I changed your script to work with my tomcat manager:

        use strict; use HTTP::Cookies; use LWP::UserAgent; # try to simulate wget: my $cookie_jar = HTTP::Cookies->new( file => "mycookie-tomcat.cookie", autosave => 1, ); my $ua = LWP::UserAgent->new; $ua->credentials ("host:8080","Tomcat Manager Application","user","pas +sword"); $ua->timeout(2); $ua->cookie_jar($cookie_jar); #$ua->env_proxy; if( !@ARGV) { warn "Requesting page"; my $request= HTTP::Request->new (GET=> "http://host:8080/manager/statu +s?XML=true"); my $status=$ua->request ($request); if ($status->is_success) { print $status->decoded_content(); } else { print STDERR $status->status_line, "\n"; return 1; } }; use Data::Dumper; warn "Cookies in jar:"; $cookie_jar->scan(sub { warn Dumper \@_ });
        And then it printout cookie when I launch without parameters:
        ... $VAR1 = [ 0, 'JSESSIONID', '1F627DA1EED0B70CC94F4D3EB9ECD378.host', '/manager', 'host.local', undef, 1, undef, undef, 1, {} ]; ...

        but file of cookie is empty:

        $ cat mycookie-tomcat.cookie #LWP-Cookies-1.0

        I discover in tcpdump / wireshark, tomcat is returning first an 401 error with "You are not authorized to view this page. If you have not changed ...", setting a cookie wich is used by lwp after (it try again the access and the page is loaded correctly), but seems like not saved in the file

        Thank you !
        PD: What is wrong with my question, so I'm getting negative reputation ?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://928054]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (10)
As of 2014-09-17 06:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (62 votes), past polls