Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re: Simulating --keep-session-cookies wget option with LWP ?

by Corion (Pope)
on Sep 27, 2011 at 09:36 UTC ( #928054=note: print w/replies, xml ) Need Help??


in reply to Simulating --keep-session-cookies wget option with LWP ?

As you don't show the code using LWP::UserAgent, it's hard for me to guess what you're doing wrong (besides that).

My guess is that you are directly overwriting the cookie file in your unshown program. I use WWW::Mechanize (not WWW::Mechandize) with (persistent) cookies and it works well. As WWW::Mechanize just uses LWP::UserAgent, both should work well with cookies.

I doubt that the cookies file that wget writes is supposed to work with LWP::UserAgent resp. HTTP::Cookies, but maybe they use the same format. I haven't seen this documented anywhere.

Replies are listed 'Best First'.
Re^2: Simulating --keep-session-cookies wget option with LWP ?
by i5513 (Pilgrim) on Sep 27, 2011 at 09:50 UTC

    I can read in Re: Passing a cookie with LWP::UserAgent about session cookies are deleted when browser (UserAgent in LWP) is closed, so every thread which I start will start with an new cookie

    My last try with HTTP::Cookies was:

    # try to simulate wget: my $cookie_jar = HTTP::Cookies->new( file => "$cookie", autosave => 1, ); my $ua = LWP::UserAgent->new ; $ua->timeout(2); $ua->cookie_jar($cookie_jar); $ua->credentials ("$host:$port","Tomcat Manager Application",$user,$pa +ssword); my $request= HTTP::Request->new (GET=> "http://$host:$port/manager/sta +tus?XML=true"); my $status=$ua->request ($request); if ($status->is_success) { $page=$status->decoded_content(); } else { print STDERR $status->status_line, "\n"; return 1; }
    Thanks!

      Your code works for me, once I fix it to actually become a program:

      use strict; use HTTP::Cookies; use LWP::UserAgent; # try to simulate wget: my $cookie_jar = HTTP::Cookies->new( file => "mycookie.cookie", autosave => 1, ); my $ua = LWP::UserAgent->new; $ua->timeout(2); $ua->cookie_jar($cookie_jar); $ua->env_proxy; if( !@ARGV) { warn "Requesting page"; my $request= HTTP::Request->new (GET=> "http://www.google.com/webhp?hl +=en"); my $status=$ua->request ($request); if ($status->is_success) { print $status->decoded_content(); } else { print STDERR $status->status_line, "\n"; return 1; } }; use Data::Dumper; warn "Cookies in jar:"; $cookie_jar->scan(sub { warn Dumper \@_ });

      It (re)generates the cookie file if run without any command line argument, and if run with a command line argument, it just dumps the cookies from the file.

      Whatever your problem seems to be, I would suspect that it lies elsewhere.

        Ok, I tested your script with google and it works.

        Then it maybe related with tomcat auth issue:
        I changed your script to work with my tomcat manager:

        use strict; use HTTP::Cookies; use LWP::UserAgent; # try to simulate wget: my $cookie_jar = HTTP::Cookies->new( file => "mycookie-tomcat.cookie", autosave => 1, ); my $ua = LWP::UserAgent->new; $ua->credentials ("host:8080","Tomcat Manager Application","user","pas +sword"); $ua->timeout(2); $ua->cookie_jar($cookie_jar); #$ua->env_proxy; if( !@ARGV) { warn "Requesting page"; my $request= HTTP::Request->new (GET=> "http://host:8080/manager/statu +s?XML=true"); my $status=$ua->request ($request); if ($status->is_success) { print $status->decoded_content(); } else { print STDERR $status->status_line, "\n"; return 1; } }; use Data::Dumper; warn "Cookies in jar:"; $cookie_jar->scan(sub { warn Dumper \@_ });
        And then it printout cookie when I launch without parameters:
        ... $VAR1 = [ 0, 'JSESSIONID', '1F627DA1EED0B70CC94F4D3EB9ECD378.host', '/manager', 'host.local', undef, 1, undef, undef, 1, {} ]; ...

        but file of cookie is empty:

        $ cat mycookie-tomcat.cookie #LWP-Cookies-1.0

        I discover in tcpdump / wireshark, tomcat is returning first an 401 error with "You are not authorized to view this page. If you have not changed ...", setting a cookie wich is used by lwp after (it try again the access and the page is loaded correctly), but seems like not saved in the file

        Thank you !
        PD: What is wrong with my question, so I'm getting negative reputation ?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://928054]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (6)
As of 2016-10-01 20:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How many different varieties (color, size, etc) of socks do you have in your sock drawer?






    Results (7 votes). Check out past polls.