Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Seeking assistance with WWW::Mechanize Form Submission

by tspang (Initiate)
on Jan 14, 2013 at 21:09 UTC ( #1013280=perlquestion: print w/ replies, xml ) Need Help??
tspang has asked for the wisdom of the Perl Monks concerning the following question:

Hello everyone - long time reader (as google turns up perlmonks quite frequently) first time poster, as I'm frustratingly stuck. The Mechanize perl doc even recommends asking for assistance here, so I'm giving it a shot.

The problem: I'm attempting to use WWW::Mechanize to log into an apc network management card to scrape the temperature data contained in one of the pages. This should be an easy task... I would think.

I started by creating a super simple form on my own to verify I could submit a form with WWW::Mechanize. I quickly got this functional.

Now, with basic functional code, I used mech-dump to get the form information that I would need to submit (and verified this by recording logging into the interface with HTTP Watch). HTTP Watch also shows that after submitting the correct username, password, and language, you recieve a 303 response directing you to a new page that contains your session ID as part of the URI - so I MUST use the data returned by the 303 to get any further into the interface.

mech-dump returned this: login_username= (text) login_password= (password) prefLanguage=00000000 (option) *00000000/English submit=Log On (submit) <NONAME>=<UNDEF> (reset)

Given this information, I modified my (functional) test script to use the data provided by mech-dump and HTTP watch:

#!/usr/bin/perl use WWW::Mechanize; use Crypt::SSLeay; my $mech = WWW::Mechanize->new( autocheck => 0); $url="https://192.168.0.2"; $mech->get( $url ); print $mech->status; $mech->form_name('HashForm1'); $mech->field("login_username",'username'); $mech->field("login_password",'password'); $mech->field("prefLanguage",00000000); $mech->submit_form(); print "form submitted, reponse was ",$mech->status," printing header\n +"; my $response = $mech->response(); for my $key ( $response->header_field_names() ) { print $key, " : ", $response->header( $key ), "\n"; } exit;
The problem at this point is, I always recieve a 500 back from the page. To make matters worse, I've tried to modify this script to submit to several other interfaces (IBM AMM, HP ILOM, etc) and none of them accept the data that I'm submitting, so it appears to be a problem with my perl.

Any help would be GREATLY appreciated, even if its just links to some extra documentation that I can read to find my own way (I've already tried the perdoc and the cookbook here: http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize/Cookbook.pod)

Thanks!

Comment on Seeking assistance with WWW::Mechanize Form Submission
Download Code
Re: Seeking assistance with WWW::Mechanize Form Submission
by Anonymous Monk on Jan 14, 2013 at 21:55 UTC
      I'll take a look at HTTP::Recorder.

      There doesn't appear to be a dump method - just dump-forms, dump-images, etc. If I set autocheck to 1 and let it exit it gives me a quite generic error:

      Server closed connection without sending any data back

      As for javascript, the only javascript on the page sets focus to the first text field. I've included it here for completeness:
      script type="text/javascript" function placeFocus() { if (document.forms.length > 0) { var field = document.forms[0]; for (i = 0; i < field.length; i++) { if ((field.elements[i].type == "text") || (field.elements[i].type == " +textarea")) { document.forms[0].elements[i].focus(); break; } } } } /script
        :) $mech->res->dump
        It is common for an image on the page to set a cookie, so HTTP::Recorder or LiveHttpHeaders or WireShark will show that if you're looking :)
Re: Seeking assistance with WWW::Mechanize Form Submission
by Gangabass (Priest) on Jan 15, 2013 at 01:09 UTC

    First I recommend you to start by checking if your form works in the browser with JavaScript disabled: remove all cookies for this site, disable JavaScript and try to submit your form.

    If the problem is not in JavaScript you need to try to add button => ... to your code. Many sites expects x and y coordinates of the submit button you click:

    $mech->submit_form( with_fields => { "login_username" =>'username', "login_password" => 'password', "prefLanguage" => '00000000', }, button => "your_submit_button_name", );
    to your code. Many sites expects x and y coordinates of the submit button you click:

      I removed all existing cookies, disabled javascript and cookies in firefox prior to logging in, and was still able to successfully log in, so it doesn't look like javascript or cookie issue (even though the site does set a cookie per HTTP Watch).

      I changed the structure of my form submission to this:

      $mech->submit_form( form_name => 'HashForm1', fields => { login_password => 'username', login_username => 'password', prefLanguage => '00000000', }, button => "submit", );
      Still to no success. Is the above code functionally different than what I had been using earlier?
      $mech->form_name('HashForm1'); $mech->field("login_username",'username'); $mech->field("login_password",'password'); $mech->field("prefLanguage",00000000); $mech->submit_form();

      Thanks for the help so far, everyone. I'm looking into HTTP:Recorder right now, and I'm feeling like I'm moving at least a little toward a solution at this point.

        Are you sure that your submit button is named "submit"? Yes, the code is different (you're now sending button info too).

        I recommend you to use LiveHttpHeaders for Firefox and make exactly same request with $mech->post()

        Thanks.
        Roman

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1013280]
Approved by Perlbotics
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (9)
As of 2014-07-25 12:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (171 votes), past polls