Perl Mechanize

zoya has asked for the wisdom of the Perl Monks concerning the following question:

Hi all, I have to fill forms on multiple webpages fetch the data parse the html into text and store it in a single file. I have the following code and every webform has different fields to be filled ,this is teh one of the website i have three more can anybody plz tell how can i do this. Suggestions are appreciated thanks.

use strict;
use warnings;
my $timeout=40;
use WWW::Mechanize;
use HTML::TreeBuilder;
use HTML::FormatText;
use HTML::Parser;

use autodie qw/ open close /;
use 5.012;
use Win32::IE::Mechanize;
use Time::HiRes 'sleep';
my $m = WWW::Mechanize->new(autocheck => 1);
my $browser = Win32::IE::Mechanize->new(visible => 1);
my $snp = "rs111";
my $content= $browser->get("http://snp-nexus.org/index.html");

my $html = $browser->content;
$browser->form_name ('snpnexus');

#$browser->field('query', 'dbsnp');
$browser->field('batch_text', 'dbsnp rs111');
$browser->tick('ensembl', "ensembl");
$browser->tick('refseq','refseq');
$browser->tick('ucsc','ucsc');
$browser->tick("sift",'sift');
$browser->tick("polyphen",'polyphen');
$browser->tick("chb",'chb');
$browser->tick("chd",'chd');
$browser->tick("tfbs",'tfbs');
$browser->tick("consv",'consv');
$browser->tick("gwas",'gwas');
$browser->tick("indel",'indel');
$browser->tick("mirbase" ,'mirbase');
$browser->tick('gad','gad');
$browser->tick('cnp' , 'cnp' );
$browser->click_button('value', 'RUN');

for (0 .. $timeout*20) {
last if $browser->{agent}->ReadyState >=5;
sleep 0.1;
}
my $html2 = $browser->content;

my $Format =HTML::FormatText->new();
my $TreeBuilder =HTML::TreeBuilder->new();
$TreeBuilder->parse($html2);
my $parsed= $Format->format($TreeBuilder);

print $parsed;
[download]

Comment on Perl Mechanize Download Code

Replies are listed 'Best First'.
Re: Perl Mechanize by runrig (Abbot) on Apr 24, 2013 at 21:12 UTC
Suggestions are appreciated thanks. Slow down. What are you having a problem with? Fetching data? Parsing HTML? Ask one question at a time, post a small amount of code that demonstrates your problem, and describe your problem (and expectations).	[reply]
Re^2: Perl Mechanize by zoya (Initiate) on Apr 25, 2013 at 06:50 UTC
My question is if i have to work with more than one website what could be the solution actually i dont lnow how to deal with multiple websites using perl mechanize. My code above is an example of what i am doing with one of the websites. I have to the same with remaining but in single code. Hopefully nw I deleivered my problem ?	[reply]
Re^3: Perl Mechanize by runrig (Abbot) on Apr 25, 2013 at 19:21 UTC
My code above is an example of what i am doing with one of the websites. I have to the same with remaining but in single code So change the url that you pass to the get() method? Also, do you want to use WWW::Mechanize, or Win32::IE::Mechanize, because you create an object for both, but only seem to use one?	[reply]
Re: Perl Mechanize by Anonymous Monk on Apr 25, 2013 at 03:33 UTC
Suggestions are appreciated thanks. Work through it, its the only way read Scraping AJAX?, get http://seleniumhq.org/ or HTTP::Recorder/Web Testing with HTTP::Recorder ... and follow the these steps and figure it out, its not hard, its just work, work you have to do See htmltreexpather.pl , Parsing HTML / Re^4: Parsing HTML, A regex question , NASA's Astronomy Picture of the Day / Re: NASA's Astronomy Picture of the Day , Re: Extracting HTML content between the h tags, Re^2: Help With Online Table Scraper, Re^4: web::scraper using an xpath, .... HTML Parser suggestions See WWW::Mechanize::Firefox::FAQ , WWW::Mechanize::FAQ, WWW::Scripter, Web::Scraper, Web::Magic, Mojo::DOM, HTTP::Recorder, Web Testing with HTTP::Recorder crossposted to http://stackoverflow.com/questions/16140778/perl-mechanize-difficulty-in-output, you should cross-link cross-postings for maximum collaboration efficiency	[reply]
Re^2: Perl Mechanize by zoya (Initiate) on Apr 25, 2013 at 06:55 UTC
Ok thank you i will check them out.	[reply]