WWW::Mechanize submission problem

[no21]

I have a working Activeperl script on my Windows box that logs in to Yahoo and downloads some HTML files.

I transferred that script to a Linux box today, and now the script does not work. Specifically, the form is not being submitted.

Activeperl script runs on:
WinXP
Perl 5.8.4
WWW-Mechanize version 0.72

Same script on Linux:
Ubuntu 5.10
Perl 5.8.7
WWW-Mechanize version 1.18


I have tried many different methods per the documentation on CPAN (click(), submit(), click_button()), and none of the methods log me in successfully.

I have Dumped the form and verified that the user and password fields are being set correctly.

The various submission methods that I tried do not die(), so I can not see why it will not work on the Linux box.

Has anyone ever run into this sort of thing before? Any help would be greatly appreciated.

[davidrw]

You could try replacing your login code w/WWW::Yahoo::Groups -- just use it to login, and then go wherever you need from there (it uses a WWW::Mechanize object underneath). This way you don't need to maintain the login part yourself:

my $y = WWW::Yahoo::Groups->new();
$y->login( $user => $pass );

# use ->get() which wraps ->agent->get():
$y->get( $your_desired_url );

# or get the agent (a WWW::Yahoo::Groups::Mechanize
# object, which is a subclass of WWW::Mechanize)
my $mech = $y->agent;
# and do whatever with it.
$mech->get( $your_desired_url );

At the least, it might help you diagnose the login problem .. (does WWW::Yahoo::Groups also fail? Does it provide any useful errors?)

[adrianh]

Has anyone ever run into this sort of thing before? Any help would be greatly appreciated.

Some code would help :-)

[no21]

I agree. Sorry about that.

This sub is what is currently working to log me in to Yahoo with ActivePerl:



sub login{

    print "Logging in to Yahoo...";

     #set url to page requiring login and password
    my $url = 'http://login.yahoo.com/config/login?.done=http://fantas
+ysports.yahoo.com&.src=spt&.intl=us';
    $mech->get( $url ) or die("Could not get $url\n");

     #these two fields were specfic to the Yahoo login HTML field name
+s 
     #for login and password.     
     
    $mech->field("login", 'my_username');
    $mech->field("passwd", 'my_password');
    $mech->click();

    print "Done.\n";

}#sub login



And the original error message was actually with the $mech->field() methods. They failed on the Linux box first. After re-reading the documentation on CPAN, I tried a different way of setting the values for those fields, and I was able to fix that problem by using this code instead of the two $mech->field lines above above:



$mech->set_visible( "my_username", "my_password");



However, apparently the submission is not working because I am not downloading the HTML content that should be there after logging in. It actually downloads an HTML page telling me that I haven't logged in.

So now my linux box code looked like this:



sub login{

    print "Logging in to Yahoo...";

     #set url to page requiring login and password
    my $url = 'http://login.yahoo.com/config/login?.done=http://fantas
+ysports.yahoo.com&.src=spt&.intl=us';
    $mech->get( $url ) or die("Could not get $url\n");

     #these two fields were specfic to the Yahoo login HTML field name
+s 
     #for login and password.     
     
    $mech->set_visible( "my_username", "my_password");
    $mech->click();

    print "Done.\n";

}#sub login



That's when I began Dumping the form to make sure the login and password fields were actually being set to the correct values, and they are.

So tried a few other submission methods other than $mech->click(). Like the following:

$mech->submit();

...and...

my $button = $mech->current_form()->find_input(undef, "submit");
$mech->click_button(input => $button)


None of these worked. Again, any help would be greatly appreciated.

[johnnywang]

I do lots of scraping/automation with LWP/WWW::Mechanize, I found the following two are very useful:
• Ethereal, so you can see what actually gets sent, and compare that with what the browser is sending.
• "perl -d", yes, perl debugger, stepping through the code is really useful.
I am able to solve most of my problems using these two (except those with heavy javascript.)

[john_oshea]

In a similar vein, the Tamper Data extension for Firefox is ridiculously handy for on-the-fly experimenting.

[johnnywang]

Since we're talking about Firefox extensions, I use the WebDeveloper extension alot, it not only can show you the headers, but also can show such things as layout outlines, which is really useful when combined with HTML::TableExtract for screen scraping. You can download it from http://chrispederick.com/work/webdeveloper/

[JamesNC]

In addition to the other comments, I would check your enviroment (HTTP_PROXY?) too, especially if you have a proxy involved on the Linux side

[planetscape]

I second johnnywang's recommendation to use Ethereal.

I have also heard good things about LiveHTTPHeaders.

HTH,

[no21]

Well, I know what I did to fix my problem, but yet I *don't* know what I did to fix my problem.

Just today I remembered the first time I tried to log on to Yahoo and retrieve HTML files. I remembered trying to install WWW::Mechanize - I think - and getting an error message at some point telling me that I needed Crypt::SSLeay.

So today I got on to CPAN and installed Crypt::SSLeay and re-ran my script, and it works now.

So I now have one more question, and one more comment.

My comment is that I still don't know why $mech->field() had to be replaced with $mech->set_visible() when running on the Linux box.

My question is, will someone explain to me why I needed Crypt::SSleay in order to get my script to work?

Thanks to everyone for their comments and assistance.

[helphand]

My question is, will someone explain to me why I needed Crypt::SSleay in order to get my script to work?

Easy, go to the url in your script;

http://login.yahoo.com/config/login?.done=http://fantas
+ysports.yahoo.com&.src=spt&.intl=us

Note the tiny note under the login box, new submits over ssl. That means you need ssl encryption for the submit, which Crypt::SSleay provides.

Scott

[monarch]

I use the following function that I wrote to log into yahoo mail and retrieve the number of messages waiting:
Read more... (4 kB)