http://www.perlmonks.org?node_id=311829

Cody Pendant has asked for the wisdom of the Perl Monks concerning the following question:

I've been trying to automate a logon to a message board.

I've tried with WWW::Automate and with LWP::UserAgent, and they should follow HTTP redirects, shouldn't they?

However when I supply the correct username and password the status code I get back from submitting the automated form (WWW:Automate) or creating the request content myself (LWP:UA) is "301 Moved permanently" and the content is just <!-- Vignette V/5 Wed Dec 03 05:07:08 2003 -->.

Any ideas what's going on?

It is possible to use this form without JavaScript by the way.



($_='kkvvttuubbooppuuiiffssqqffssmmiibbddllffss') =~y~b-v~a-z~s; print

Replies are listed 'Best First'.
Re: Can't Automate Login To System
by edoc (Chaplain) on Dec 03, 2003 at 05:19 UTC

    are you 'POST'ing the data to the other server? If so you will need to enable POST requests to be redirectable with

    push @{ $ua->requests_redirectable }, 'POST';

    as per the LWP::Useragent docs.

    cheers,

    J

      Thanks, I had looked at the docs and totally missed that.

      Hasn't fixed my problem though. Now I get "405: method not allowed", the method POST is not allowed for (URL), but that's not the URL I'm trying to submit to, it's the original page with the login form.

      The form itself is a little strange, the action of the form is set to a vignette-looking HTML url.

      Anyone want to look at the form, at http://users.guardian.co.uk/signin/0,12930,-5,00.html and tell me what might be going wrong?

      I should say again, the form can be submitted without JavaScript, so what the point of their weird MD5-hashing of hidden fields and passwords is I don't know.



      ($_='kkvvttuubbooppuuiiffssqqffssmmiibbddllffss') =~y~b-v~a-z~s; print

        I should say again, the form can be submitted without JavaScript
        Your question looked pretty interesting, so I took a stab at it. Turns out I can't log in at all using mozilla.
        Here's what I tried:
        • Turning JavaScript on/off
        • Making sure JS has access to create/read cookies
        • Masquerading as IE6 on WinXP (ain't the prefbar wonderful?)
        • Using IE6 on Win2k

        Of these, only using genuine IE worked — they're doing something nefarious.
        So, a quick experiment with an HTTP Sniffer reveals that IE POSTs the request, to which the webserver doesn't appear to respond, but it does offer a 301 redirect to another GET request.

        I've got absolutely no idea how this works — as I understand HTTP, this shouldn't happen; which probably explains why it doesn't work with Mozilla. Could you show us some (password/username-sanitized) code so we can play with it ourselves?

        Update: This bit added

        so what the point of their weird MD5-hashing of hidden fields and passwords is I don't know
        That appears to be used if JavaScript's enabled (which would probably be for 95% of their users) — it prevents transmission of the password in cleartext. Instead, an MD5 hash of their password with a server-provided challenge is sent. The challenge token probably (hopefully) expires once used, and after a timeout period. This is a pretty effective way of preventing password-sniffing, and because the password entry would still be there for non-JavaScript browsers, it'd work for users without JavaScript (although they'd have to be using a broken browser as discussed above).

        cheers


        davis
        It's not easy to juggle a pregnant wife and a troubled child, but somehow I managed to fit in eight hours of TV a day.

Re: Can't Automate Login To System
by jmanning2k (Pilgrim) on Dec 03, 2003 at 20:52 UTC
    If POST is denied, GET is probably correct.

    When I last did this, I had to allow session cookies to get a authorization page to remember my login. See HTTP::Cookies

    use LWP::UserAgent; use HTTP::Cookies; my $ua = new LWP::UserAgent(); $ua->cookie_jar(HTTP::Cookies->new());
    Then, do your login and it should get you past the first redirect.

    (You may have to follow that redirect to get to the first page. simple_request doesn't do that, but WWW::Automate might. In either case, I just took the redirect to the homepage rather than too the login page as a sign of successful authorization and skipped right to the rest of my code.)

    ~J

      Thanks, but I'm already allowing cookies, and they're being successfully created, so that's not the answer I'm afraid.

      My next strategy was going to be log in using IE, then take the cookies it set for IE and manually add them to the LWP cookies file. Or should I just give up?



      ($_='kkvvttuubbooppuuiiffssqqffssmmiibbddllffss') =~y~b-v~a-z~s; print

        Sorry, you didn't mention that before, and it certainly gave me trouble earlier. Just checking.

        Stealing the cookies from IE is worth a try. It should work until they expire or are logged out.

        Then again, that simply avoids the problem and pre-authorizes you. Not much of a fix to your original problem.

Re: Can't Automate Login To System
by Cody Pendant (Prior) on Dec 05, 2003 at 11:39 UTC
    For the record, antirice came up with some code which succeeded. I'm very grateful. I haven't quite figured out what the trick is myself, but both he and I have the code so if you'd like to see it, ask.


    ($_='kkvvttuubbooppuuiiffssqqffssmmiibbddllffss') =~y~b-v~a-z~s; print