Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

WWW::Mechanize submission problem

by no21 (Sexton)
on Feb 05, 2006 at 06:12 UTC ( #528041=perlquestion: print w/replies, xml ) Need Help??

no21 has asked for the wisdom of the Perl Monks concerning the following question:

I have a working Activeperl script on my Windows box that logs in to Yahoo and downloads some HTML files.

I transferred that script to a Linux box today, and now the script does not work. Specifically, the form is not being submitted.

Activeperl script runs on:
WinXP
Perl 5.8.4
WWW-Mechanize version 0.72

Same script on Linux:
Ubuntu 5.10
Perl 5.8.7
WWW-Mechanize version 1.18


I have tried many different methods per the documentation on CPAN (click(), submit(), click_button()), and none of the methods log me in successfully.

I have Dumped the form and verified that the user and password fields are being set correctly.

The various submission methods that I tried do not die(), so I can not see why it will not work on the Linux box.

Has anyone ever run into this sort of thing before? Any help would be greatly appreciated.

Replies are listed 'Best First'.
Re: WWW::Mechanize submission problem
by davidrw (Prior) on Feb 05, 2006 at 15:06 UTC
    You could try replacing your login code w/WWW::Yahoo::Groups -- just use it to login, and then go wherever you need from there (it uses a WWW::Mechanize object underneath). This way you don't need to maintain the login part yourself:
    my $y = WWW::Yahoo::Groups->new(); $y->login( $user => $pass ); # use ->get() which wraps ->agent->get(): $y->get( $your_desired_url ); # or get the agent (a WWW::Yahoo::Groups::Mechanize # object, which is a subclass of WWW::Mechanize) my $mech = $y->agent; # and do whatever with it. $mech->get( $your_desired_url );
    At the least, it might help you diagnose the login problem .. (does WWW::Yahoo::Groups also fail? Does it provide any useful errors?)
Re: WWW::Mechanize submission problem
by adrianh (Chancellor) on Feb 05, 2006 at 11:50 UTC
    Has anyone ever run into this sort of thing before? Any help would be greatly appreciated.

    Some code would help :-)

      I agree. Sorry about that.

      This sub is what is currently working to log me in to Yahoo with ActivePerl:

      sub login{ print "Logging in to Yahoo..."; #set url to page requiring login and password my $url = 'http://login.yahoo.com/config/login?.done=http://fantas +ysports.yahoo.com&.src=spt&.intl=us'; $mech->get( $url ) or die("Could not get $url\n"); #these two fields were specfic to the Yahoo login HTML field name +s #for login and password. $mech->field("login", 'my_username'); $mech->field("passwd", 'my_password'); $mech->click(); print "Done.\n"; }#sub login


      And the original error message was actually with the $mech->field() methods. They failed on the Linux box first. After re-reading the documentation on CPAN, I tried a different way of setting the values for those fields, and I was able to fix that problem by using this code instead of the two $mech->field lines above above:

      $mech->set_visible( "my_username", "my_password");


      However, apparently the submission is not working because I am not downloading the HTML content that should be there after logging in. It actually downloads an HTML page telling me that I haven't logged in.

      So now my linux box code looked like this:

      sub login{ print "Logging in to Yahoo..."; #set url to page requiring login and password my $url = 'http://login.yahoo.com/config/login?.done=http://fantas +ysports.yahoo.com&.src=spt&.intl=us'; $mech->get( $url ) or die("Could not get $url\n"); #these two fields were specfic to the Yahoo login HTML field name +s #for login and password. $mech->set_visible( "my_username", "my_password"); $mech->click(); print "Done.\n"; }#sub login


      That's when I began Dumping the form to make sure the login and password fields were actually being set to the correct values, and they are.

      So tried a few other submission methods other than $mech->click(). Like the following:
      $mech->submit();
      ...and...
      my $button = $mech->current_form()->find_input(undef, "submit"); $mech->click_button(input => $button)

      None of these worked. Again, any help would be greatly appreciated.

Re: WWW::Mechanize submission problem
by johnnywang (Priest) on Feb 05, 2006 at 20:03 UTC
    I do lots of scraping/automation with LWP/WWW::Mechanize, I found the following two are very useful:
    • Ethereal, so you can see what actually gets sent, and compare that with what the browser is sending.
    • "perl -d", yes, perl debugger, stepping through the code is really useful.
    I am able to solve most of my problems using these two (except those with heavy javascript.)
        Since we're talking about Firefox extensions, I use the WebDeveloper extension alot, it not only can show you the headers, but also can show such things as layout outlines, which is really useful when combined with HTML::TableExtract for screen scraping. You can download it from http://chrispederick.com/work/webdeveloper/
Re: WWW::Mechanize submission problem
by JamesNC (Chaplain) on Feb 05, 2006 at 19:20 UTC
    In addition to the other comments, I would check your enviroment (HTTP_PROXY?) too, especially if you have a proxy involved on the Linux side
Re: WWW::Mechanize submission problem
by planetscape (Chancellor) on Feb 06, 2006 at 02:01 UTC
Re: WWW::Mechanize submission problem
by no21 (Sexton) on Feb 06, 2006 at 04:35 UTC
    Well, I know what I did to fix my problem, but yet I *don't* know what I did to fix my problem.

    Just today I remembered the first time I tried to log on to Yahoo and retrieve HTML files. I remembered trying to install WWW::Mechanize - I think - and getting an error message at some point telling me that I needed Crypt::SSLeay.

    So today I got on to CPAN and installed Crypt::SSLeay and re-ran my script, and it works now.

    So I now have one more question, and one more comment.

    My comment is that I still don't know why $mech->field() had to be replaced with $mech->set_visible() when running on the Linux box.

    My question is, will someone explain to me why I needed Crypt::SSleay in order to get my script to work?

    Thanks to everyone for their comments and assistance.

      My question is, will someone explain to me why I needed Crypt::SSleay in order to get my script to work?

      Easy, go to the url in your script;

      http://login.yahoo.com/config/login?.done=http://fantas +ysports.yahoo.com&.src=spt&.intl=us

      Note the tiny note under the login box, new submits over ssl. That means you need ssl encryption for the submit, which Crypt::SSleay provides.

      Scott

Re: WWW::Mechanize submission problem
by monarch (Priest) on Feb 06, 2006 at 03:55 UTC
    I use the following function that I wrote to log into yahoo mail and retrieve the number of messages waiting:

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://528041]
Approved by willyyam
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (7)
As of 2021-03-03 09:45 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My favorite kind of desktop background is:











    Results (76 votes). Check out past polls.

    Notices?