Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

In a Web Page forms are displayed as empty

by jesuashok (Curate)
on Mar 30, 2007 at 10:27 UTC ( #607429=perlquestion: print w/ replies, xml ) Need Help??
jesuashok has asked for the wisdom of the Perl Monks concerning the following question:

monks,

I am using WWW::Mechanize for my inhouse project. Earlier I asked some question in WWW::Mechanize - Could not connect to a server and I got clarification from that node replies. Also there is a recent node which discusses about web forms in Submit HTML Forms with WWW:Mechanize. I picked up some code from there and used in my code, but I did not get any from names.

#!/usr/bin/perl use WWW::Mechanize ; use LWP::Debug qw(+); use strict; use warnings; use Data::Dumper; #use LWP::UserAgent; my $mech = WWW::Mechanize->new ; #my $ua = LWP::UserAgent->new; $mech->proxy(['http','ftp'], 'http://proxy_ip_dress:port_number/'); $mech ->get("http://application_ip_address:8082/root/Login") ; if ( $mech ->success ) { print "Connection Success ", $mech ->res->st +atus_line, "\n"; } else { print "Status :" , $mech ->res->status_line +, "\n"; } my @webforms = $mech->forms(); print Dumper [ @webforms ]; # @webforms is empty foreach my $form (@webforms) { my @inputfields = $form->param; foreach my $inputfield (@inputfields) { print Dumper [ $inputfield ]; } print Dumper [ $form ]; }
If I try to get google page I am getting value in the webforms arrary. If I use my application URL I am not getting that. Am I missing still something in the code ? will there be any web page without web forms ? Just curious to know.



hmmm ....let me think what did I said

Comment on In a Web Page forms are displayed as empty
Download Code
Re: In a Web Page forms are displayed as empty
by Preceptor (Chaplain) on Mar 30, 2007 at 10:52 UTC
    Well, if you're not getting any errors, then ... yes, you should have some content.

    If you're looking at just checking it works, is there a reason you're not using $mech -> content to see what you're actually fetching? I'd consider it more likely that something's up with your forms processing, than something trivial like a web page fetch...

      Hi Preceptor,

      Sorry for the delayed comment.
      As you suggested I used $mech -> content. I got the following output.

      #!/usr/bin/perl use WWW::Mechanize ; use LWP::Debug qw(+); use strict; use warnings; use Data::Dumper; my $mech = WWW::Mechanize->new ; $mech->proxy(['http','ftp'], 'http://63.117.9.150:8080/'); #Fetch URL or Die Tryin' $mech ->get("http://<application_ip>:8082/metrica_root/Login") ; if ( $mech ->success ) { print "Connection Success ", $mech ->res->sta +tus_line, "\n"; } else { print "Status :" , $mech ->res->status_line , + "\n"; } my @webforms = $mech->forms(); print Dumper [ @webforms ]; print "Trying to print the content ......\n"; $mech->content; foreach my $form (@webforms) { my @inputfields = $form->param; foreach my $inputfield (@inputfields) { print Dumper [ $inputfield ]; } print Dumper [ $form ]; }
      Output :- LWP::UserAgent::new: () LWP::UserAgent::proxy: ARRAY(0x1bf0e28) http://63.117.9.150:8080/ LWP::UserAgent::proxy: http http://63.117.9.150:8080/ LWP::UserAgent::proxy: ftp http://63.117.9.150:8080/ LWP::UserAgent::request: () HTTP::Cookies::add_cookie_header: Checking 155.226.230.173 for cookies HTTP::Cookies::add_cookie_header: Checking .226.230.173 for cookies HTTP::Cookies::add_cookie_header: Checking 226.230.173 for cookies HTTP::Cookies::add_cookie_header: Checking .230.173 for cookies HTTP::Cookies::add_cookie_header: Checking 230.173 for cookies HTTP::Cookies::add_cookie_header: Checking .173 for cookies LWP::UserAgent::send_request: GET http://155.226.230.173:8082/metrica_ +root/Login LWP::UserAgent::_need_proxy: Proxied to http://63.117.9.150:8080/ LWP::Protocol::http::request: () LWP::Protocol::collect: read 266 bytes LWP::Protocol::collect: read 301 bytes HTTP::Cookies::extract_cookies: Set cookie JSESSIONID => 0000000000000 +01a49444c3 a687474702f52657150726f636573736f723a312e30000000000000010000000000000 +0ac0001020 0000000103135352e3232362e3233302e31373300c39300000000004900504d4300000 +0000000001 a49444c3a687474702f52657150726f636573736f723a312e300020200000000974635 +f696e73743 1002020200000000d2f746f6d636174345f706f6100000000000000035649530300000 +0050005070 17f0000000000000000000008000000005649530000000001000000140000000000010 +0010000000 00001010900000000_2925080BF3F637D984812BF731798941 LWP::UserAgent::request: Simple response: OK Connection Success 200 OK $VAR1 = []; Trying to print the content ......
      I don't have any clues to move further on this. I am expecting your valid inputs on the same.
        Hi jesuashok,

        I had a hunch that the pages in question are generated by JavaScript using 'document.write', so I /msg'd you to confirm this. I have seen some pages that use document.write to output html. Let us take for a basic example the following html page:
        <html> <head> <title>Bad Example</title> </head> <body> <script> document.write('<H1>JavaScript made me do it!</H1>') </script> </body> </html>
        Now the following Perl code:
        #!/usr/bin/perl use strict; use warnings; use WWW::Mechanize; my $mech = WWW::Mechanize->new; $mech->get('http://yourserverhere/test.html'); print $mech->content( format => "text" );
        Which outputs: "Bad Example". Why? Because "Please note that Mech does NOT support JavaScript. Please check the FAQ in WWW::Mechanize::FAQ for more.".

        You can use Win32::IE::Mechanize or Mozilla::Mechanize or alike to control a browser which does know how to deal with JavaScript. This is another reason to always refer to the documentation.

        Update: Excuse the lack of error checking in the Perl script above, I wanted to get this post to you ASAP and I have to attend a project meeting.

        Hope this helps

        Martin
Re: In a Web Page forms are displayed as empty
by MonkE (Hermit) on Mar 30, 2007 at 13:06 UTC
    A related technique I've used before is to put the content in a file with $mech->save_content('filename.html'). Then have a look at it with a browser using a file:// URL.
      Hi

      #!/usr/bin/perl use WWW::Mechanize ; use LWP::Debug qw(+); use strict; use warnings; use Data::Dumper; #use LWP::UserAgent; my $mech = WWW::Mechanize->new ; #my $ua = LWP::UserAgent->new; $mech->proxy(['http','ftp'], 'http://proxy_ip_dress:port_number/'); $mech ->get("http://application_ip_address:8082/root/Login") ; if ( $mech ->success ) { print "Connection Success ", $mech ->res->st +atus_line, "\n"; } else { print "Status :" , $mech ->res->status_line +, "\n"; } $mech->save_content('sample.html');
      I have executed the above code and saved the content of a page in "sample.html". when I opened that file in Browser I could a see page saying that Page Cannot be displyed.
      any other points would be appreciated.



      hmmm ....let me think what did I said

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://607429]
Approved by GrandFather
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (9)
As of 2015-07-03 12:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (53 votes), past polls