ASPX Form Post

[LeoCoder]

Hello Monks - I am trying to retrieve the table data from this page http://bogc.dnrc.mt.gov/WebApps/DataMiner/Wells/WellCompletions.aspx using mechanize. I am not able to get the result page after post. I am passing all parameters with the form. But it is not working. Please assists me to solve this invisible bug. :)

use strict;
use warnings;
use LWP::Simple;
use WWW::Mechanize;

#Access to the root page
    my $agent;
    $agent = WWW::Mechanize->new();
    my $rootpage = "http://bogc.dnrc.mt.gov/WebApps/DataMiner/Wells/We
+llCompletions.aspx";

#Get the root page
    $agent->get($rootpage);

#Get Form data
    $agent->form_name('aspnetForm');
    my $eventtarget = $agent->field('__EVENTTARGET');
    my $eventarg = $agent->field('__EVENTARGUMENT');
    my $lastfocus = $agent->field('__LASTFOCUS');
    my $viewstate = $agent->field('__VIEWSTATE');
    my $viewstategen = $agent->field('__VIEWSTATEGENERATOR');
    my $eventValidation = $agent->field('__EVENTVALIDATION');
    
#Construct fields 
    my $fields =  {
        'ctl00$ScriptManager1' => 'ctl00$SiteContentPlaceHolder$FormFi
+lterNoNav1$UpdatePanelFormFilterNoNav|ctl00$SiteContentPlaceHolder$Fo
+rmFilterNoNav1$btnSearch',
        '__EVENTTARGET' => $eventtarget,
        '__EVENTARGUMENT' => $eventarg,
        '__LASTFOCUS' => $lastfocus,
        '__VIEWSTATE' => $viewstate,
        '__VIEWSTATEGENERATOR' => $viewstategen,
        '__EVENTVALIDATION' => $eventValidation,
        'ctl00$SiteContentPlaceHolder$FormFilterNoNav1$cboFilterOption
+' => 'Dt_Effect|date',
        'ctl00$SiteContentPlaceHolder$FormFilterNoNav1$cboFilterType' 
+=> '<',
        'ctl00$SiteContentPlaceHolder$FormFilterNoNav1$txtFilter' => '
+09/06/2016',
        'ctl00$SiteContentPlaceHolder$FormFilterNoNav1$txtFilterString
+' => '',
        'ctl00$SiteContentPlaceHolder$FormFilterNoNav1$btnSearch' => '
+Search'
    };
#Submit request to get data.
    $result = $agent->submit_form(
                form_name => 'aspnetForm',
                fields => $fields,
                );
    $agent->save_content("C:/" . "Completions.html");

[choroba]

Is JavaScript involved in the submission? WWW::Mechanize doesn't support it, as documented.

[LeoCoder]

Thanks. I am not sure about java script. Is there any other package that replaces mechanize and have all feature for handling web request. Additional information: java script are used to fill the form parameter during page load. I am getting those parameter results and passing them again while posting.

[Corion]

If the page needs Javascript, you will need to find a version of WWW::Mechanize that supports Javascript. See for example Win32::IEAutomation, WWW::Mechanize::Firefox or WWW::Mechanize::PhantomJs.

Alternatively, find out what the Javascript does and do the same with Perl instead of Javascript.

[hippo]

But it is not working.

In which case the problem is probably somewhere between "your computer isn't switched on" and "your ISP is filtering your traffic".

Please assists me to solve this invisible bug. :)

Please assist us to assist you by at least describing the symptoms in detail. Post the exact text of the error message received. Say which versions of perl and the 2 modules you are using (and maybe explain why you are using LWP::Simple when the rest of your code doesn't mention it). Say which O/S you are running. In essence, give some detail. Thanks.

[LeoCoder]

Let me explain the problem. I am trying to extract the table data from that website. It requires current date in the textbox and click search. I used Firefox to inspect the network data that are passed to server. using that data, I constructed the parameters manually and submit the form. but, server is not throwing any error and result. I have shared the code here to get some thoughts from perlmonks about whether anything is missing in this code or what should i do to scrape ASPX webpage. I am new to Perl programming. My system details Windows, ActiveState Perl 4.14.

[hippo]

Thanks for elaborating. Unfortunately the only extra information here about what actually goes wrong is this bit:

but, server is not throwing any error and result

which isn't very clear because while it says what doesn't happen, it fails to mention what does actually happen. Does the script hang? Does it die? Does it silently finish? What did you do to try to find the root cause? Have you been through the Basic debugging checklist?

My system details Windows, ActiveState Perl 4.14.

That's an almost unfeasibly old version of perl and certainly the code which you have posted would not even compile on such an ancient version. Did you mean 5.14 instead?