In a Web Page forms are displayed as empty

[jesuashok]

monks,

I am using WWW::Mechanize for my inhouse project. Earlier I asked some question in WWW::Mechanize - Could not connect to a server and I got clarification from that node replies. Also there is a recent node which discusses about web forms in Submit HTML Forms with WWW:Mechanize. I picked up some code from there and used in my code, but I did not get any from names.

#!/usr/bin/perl
use WWW::Mechanize ;
use LWP::Debug qw(+);
use strict;
use warnings;
use Data::Dumper;
#use LWP::UserAgent;

my $mech = WWW::Mechanize->new ;
#my $ua = LWP::UserAgent->new;
$mech->proxy(['http','ftp'], 'http://proxy_ip_dress:port_number/');

 $mech ->get("http://application_ip_address:8082/root/Login") ;

 if ( $mech ->success ) { print "Connection Success ", $mech ->res->st
+atus_line, "\n"; } 
 else                   { print "Status :" , $mech ->res->status_line 
+, "\n"; }



my @webforms = $mech->forms();
print Dumper [ @webforms ]; # @webforms is empty

foreach my $form (@webforms) {
   my @inputfields = $form->param;
   
   foreach my $inputfield (@inputfields) {
      print Dumper [ $inputfield ];
   }
   
   print Dumper [ $form ];
}

If I try to get google page I am getting value in the webforms arrary. If I use my application URL I am not getting that. Am I missing still something in the code ? will there be any web page without web forms ? Just curious to know.

[Preceptor]

Well, if you're not getting any errors, then ... yes, you should have some content.

If you're looking at just checking it works, is there a reason you're not using $mech -> content to see what you're actually fetching? I'd consider it more likely that something's up with your forms processing, than something trivial like a web page fetch...

[jesuashok]

Hi Preceptor,

Sorry for the delayed comment.
As you suggested I used $mech -> content. I got the following output.

#!/usr/bin/perl
use WWW::Mechanize ;
use LWP::Debug qw(+);
use strict;
use warnings;
use Data::Dumper;

my $mech = WWW::Mechanize->new ;
$mech->proxy(['http','ftp'], 'http://63.117.9.150:8080/');

#Fetch URL or Die Tryin'
$mech ->get("http://<application_ip>:8082/metrica_root/Login") ;

if ( $mech ->success ) { print "Connection Success ", $mech ->res->sta
+tus_line, "\n"; } 
else                   { print "Status :" , $mech ->res->status_line ,
+ "\n"; }

my @webforms = $mech->forms();
print Dumper [ @webforms ];

print "Trying to print the content ......\n";
$mech->content;

foreach my $form (@webforms) {
   my @inputfields = $form->param;
   
   foreach my $inputfield (@inputfields) {
      print Dumper [ $inputfield ];
   }
   
   print Dumper [ $form ];
}

Output :-
LWP::UserAgent::new: ()
LWP::UserAgent::proxy: ARRAY(0x1bf0e28) http://63.117.9.150:8080/
LWP::UserAgent::proxy: http http://63.117.9.150:8080/
LWP::UserAgent::proxy: ftp http://63.117.9.150:8080/
LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking 155.226.230.173 for cookies
HTTP::Cookies::add_cookie_header: Checking .226.230.173 for cookies
HTTP::Cookies::add_cookie_header: Checking 226.230.173 for cookies
HTTP::Cookies::add_cookie_header: Checking .230.173 for cookies
HTTP::Cookies::add_cookie_header: Checking 230.173 for cookies
HTTP::Cookies::add_cookie_header: Checking .173 for cookies
LWP::UserAgent::send_request: GET http://155.226.230.173:8082/metrica_
+root/Login

LWP::UserAgent::_need_proxy: Proxied to http://63.117.9.150:8080/
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 266 bytes
LWP::Protocol::collect: read 301 bytes
HTTP::Cookies::extract_cookies: Set cookie JSESSIONID => 0000000000000
+01a49444c3
a687474702f52657150726f636573736f723a312e30000000000000010000000000000
+0ac0001020
0000000103135352e3232362e3233302e31373300c39300000000004900504d4300000
+0000000001
a49444c3a687474702f52657150726f636573736f723a312e300020200000000974635
+f696e73743
1002020200000000d2f746f6d636174345f706f6100000000000000035649530300000
+0050005070
17f0000000000000000000008000000005649530000000001000000140000000000010
+0010000000
00001010900000000_2925080BF3F637D984812BF731798941
LWP::UserAgent::request: Simple response: OK
Connection Success 200 OK
$VAR1 = [];
Trying to print the content ......

I don't have any clues to move further on this. I am expecting your valid inputs on the same.

[marto]

Hi jesuashok,

I had a hunch that the pages in question are generated by JavaScript using 'document.write', so I /msg'd you to confirm this. I have seen some pages that use document.write to output html. Let us take for a basic example the following html page:

<html>
<head>
<title>Bad Example</title>
</head>
<body>
<script>
document.write('<H1>JavaScript made me do it!</H1>')
</script>
</body>
</html>

Now the following Perl code:

#!/usr/bin/perl

use strict;
use warnings;

use WWW::Mechanize;

my $mech = WWW::Mechanize->new;

$mech->get('http://yourserverhere/test.html');

print $mech->content( format => "text" );

Which outputs: "Bad Example". Why? Because "Please note that Mech does NOT support JavaScript. Please check the FAQ in WWW::Mechanize::FAQ for more.".

You can use Win32::IE::Mechanize or Mozilla::Mechanize or alike to control a browser which does know how to deal with JavaScript. This is another reason to always refer to the documentation.

Update: Excuse the lack of error checking in the Perl script above, I wanted to get this post to you ASAP and I have to attend a project meeting.

Hope this helps

Martin

[MonkE]

A related technique I've used before is to put the content in a file with $mech->save_content('filename.html'). Then have a look at it with a browser using a file:// URL.