Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

LWP, extract_cookies, etc.

by inblosam (Monk)
on May 22, 2002 at 06:00 UTC ( [id://168363] : perlquestion . print w/replies, xml ) Need Help??

inblosam has asked for the wisdom of the Perl Monks concerning the following question:

I have taken the advice of many wise perlmonks and modified my original troublesome script (see my post). I now get a new error that I can't seem to overcome. The error is:

Can't locate object method "url" via package "HTTP::Headers" (perhaps you forgot to load "HTTP::Headers"?) at (eval 43) line 1.

So now I figure there is a problem, but can't seem to locate it. Do I have the variables and add_cookie_header/extract_cookie methods right? See my new and improved (hopefully) code below. THANKS!!

#!/usr/lib/perl use LWP::UserAgent; use HTTP::Cookies; use HTTP::Request::Common; use HTTP::Headers; #is this where I declare cookie_jar?? seems like I am doing it twice my $cookie_jar = HTTP::Cookies->new; my $ua = LWP::UserAgent->new; #but it seems like I am nearly doing the same thing here #but I need the variable below for add_cookie_header and extract_cooki +es $ua->cookie_jar(HTTP::Cookies->new(file => 'cookie_jar', autosave =>1) +); my $request = $ua->simple_request(POST " +n.cfm", { username =>'abcuser', userpass =>'abc123', submit =>'Submit' }); while ($request->is_redirect) { my $u = $request->header('location') or die "missing location: ", $request->as_string; #print "redirecting to $u\n"; $u=~s/^\.+/http\:\/\/www\.sitedomain\.com/s; $request = $ua->simple_request(GET $u); } #do this before extract? is that the right variable to do it on? #HTTP::Cookies says that add_cookie_header must have a "valid url attr +ibute". #that lingo doesn't click with me $cookie_jar->add_cookie_header($request); my $response = $ua->request($request); $cookie_jar->extract_cookies($response); $ua->cookie_jar($cookie_jar); my $request = $ua->simple_request(POST " +.cfm", { firstname =>"Michael", lastname =>"Jensen", company =>"companyname", address =>"111 East 222 South", city =>"Provo", state =>"UT", zip =>"99999" }); if ($request->is_success) { $a = "worked"; } else{ $a = "failed"; } print "$a";

Michael Jensen
InShift Technologies

Replies are listed 'Best First'.
Re: LWP, extract_cookies, etc.
by IlyaM (Parson) on May 22, 2002 at 08:47 UTC
    The problem in you code is that you mixing HTTP::Response and HTTP::Request objects. Method call $ua->simple_request returns HTTP::Response object which you assign to variable $request. Until line
    you use this object correctly (though variable name is very misleading). On this line your program crashes because $cookie_jar->add_cookie_header expects HTTP::Request object instead of HTTP::Response. Next line also is not going to work because $ua->request also expects HTTP::Request object.

    Well, this is the reasons why it crashes. Anyway I'm not sure what are you trying to achive. Seems that you just want to make two POST requests. Why do you want to handle redirects yourself? LWP can do it itself. Why do you want to handle cookies yourself? LWP again can do it itself. Let me show my version of this code.

    #!/usr/lib/perl # be good boy and always do following use strict; use warnings; # you don't have to use other HTTP modules because # LWP::UserAgent uses them itself use LWP::UserAgent; use HTTP::Request::Common; use HTTP::Cookies; my $ua = LWP::UserAgent->new; # this method call creates cookie jar object and tells # LWP::UserAgent object to use it automatically. You do not # have to call extract_cookies or add_cookie_headers # yourself $ua->cookie_jar(HTTP::Cookies->new(file => 'cookie_jar', autosave =>1) +; # configure LWP::UserAgent to follow redirects after POST push @{ $ua->requests_redirectable }, 'POST'; # first POST request - no need to handle redirects youself. # Let LWP handle them automatically. This is why I use # 'request' instead of 'simple_request'. $ua->request(POST " +n.cfm", { username =>'abcuser', userpass =>'abc123', submit =>'Submit' }); # Second POST request - same as in your code my $request = $ua->request(POST "", { firstname =>"Michael", lastname =>"Jensen", company =>"companyname", address =>"111 East 222 South", city =>"Provo", state =>"UT", zip =>"99999" }); print $request->is_success ? "worked\n" : "failed\n";
    BTW are you trying to write automated test for login form? Consider using web testing modules like HTTP::WebTest or HTTP::Monkeywrench.

    P.S. And of course let's repeat it again: Use strict and warnings!

    Update: s/simple_request/request/, added code to turn on handling of redirects after POST in LWP::UserAgent, added use HTTP::Cookies.

    Update: added missing ';'

    Ilya Martynov (

      Very helpful, but now I have that crazy HTTP 302 error. I had to add use HTTP::Cookies; into the script again because it didn't like not having it.

      Any ideas why I get the error and/or how do I get around it? Also, Ilya you had simple_request on the second post. Did you mean to have just 'request' there? And do I not need "my $request = " on the first "$ua->request(POST..." line? Or the "$ua->cookie_jar" line?

      What I am trying to accomplish is to update my user information on a website that requires a login. So first I need to login and then use the cookie info for posting my user information on a different form. THANKS!

      #!/usr/lib/perl -w use strict; use LWP::UserAgent; use HTTP::Request::Common; # didn't work without this use statement use HTTP::Cookies; my $ua = LWP::UserAgent->new; $ua->cookie_jar(HTTP::Cookies->new(file => 'cookie_jar', autosave =>1) +); $ua->request(POST "", { username =>'abcuser', userpass =>'abc123', submit =>'Submit' }); my $request = $ua->request(POST " +e.cfm", { firstname =>"MyName", lastname =>"MyLastName", company =>"companyname", address =>"111 East 222 South", city =>"Provo", state =>"UT", zip =>"99999" }); print $request->is_success ? "worked\n" : "failed\n"; # this shows me that I have the following error: # HTTP/1.1 302 (Found) Object Moved... print $request->as_string;
        Very helpful, but now I have that crazy HTTP 302 error.

        The problem with HTTP error is caused by default behaviour of LWP::UserAgent. Sorry I forgot about it when I wrote my version of this script. LWP::UserAgent by default doesn't handles redirects after POST requests automatically. If you have not very old version of LWP::UserAgent you can redifine this behaviour using following sniplet:

        push @{ $ua->requests_redirectable }, 'POST';

        Also, Ilya you had simple_request on the second post. Did you mean to have just 'request' there?

        I'm sorry. I did mean $ua->request instead of $ua->simple_request.

        And do I not need "my $request = " on the first "$ua->request(POST..." line?

        Since you do not use response object from first request there is no need to store it in any variable.

        I have updated script in my first node with all fixes.

        Ilya Martynov (

Re: LWP, extract_cookies, etc.
by tadman (Prior) on May 22, 2002 at 08:10 UTC
    Your other post featured the use of -w, which is great, but there's no use strict, and that can be dangerous. When debugging, it can sure save your bacon, so make it a habit. It's mean, it complains a lot, but that's just tough love.

    If you make a $cookie_jar then you can supply that to your function like $ua->cookie_jar($cookie_jar) instead of doing it twice, as you have questioned yourself doing, though you will want to make sure the $cookie_jar is initialized with the parameters in your second instance.

    IlyaM points out a feature of HTTP::Request::Common below which I mistook for an error. Never had any use for these before, I suppose, and bracketless function calls are hard to parse at times.
      $ua->simple_request(POST "http://...") is correct Perl code. Have you noticed use HTTP::Request::Common; at the begining of them script ? It exports couple of handy subroutines like GET, POST and HEAD which return HTTP::Request object. See documentation for HTTP::Request::Common.

      Ilya Martynov (