http://www.perlmonks.org?node_id=996430

DamianMyers has asked for the wisdom of the Perl Monks concerning the following question:

Hello PerlMonks,

I have been working with WWW::Mechanize to automatically fill out forms.
However, I am bumped into an issue regarding JavaScript. I have been trying
to automatically fill in Dell's web portal for filing hardware issues.

I am unable to process the pages because they are using JavaScript to mask the
URL's, can anyone provide any guidance on getting around this?

Replies are listed 'Best First'.
Re: Perl auto completing forms
by davido (Cardinal) on Sep 30, 2012 at 01:04 UTC

    Perl is huge. It's used in a lot of diverse places. Web scraping is huge. It's done all the time. Javascript is huge. It's ubiquitous in the context of the web. WWW::Mechanize doesn't directly handle Javascript. The number of people who have encountered this issue is also huge. It's a highly searchable topic that has been answered many times -- a FAQ (which might explain the beating you're taking in votes right now). And it's also why the POD for WWW::Mechanize has the following:

    Please note that Mech does NOT support JavaScript, you need additional software for that. Please check "JavaScript" in WWW::Mechanize::FAQ for more.

    In the WWW::Mechanize::FAQ you will find a section titled "JavaScript". There's a good explanation there on how it may be possible to simply provide the proper get/post requests instead of relying on JavaScript parsing to deal with the website. But it also has a subcategory, "Which modules work like Mechanize and have JavaScript support?". You'll find some good suggestions there on how to proceed.


    Dave

Re: Perl auto completing forms
by ckj (Chaplain) on Sep 30, 2012 at 05:56 UTC

    First of all to scrape javascript kind of site, you can't do it using WWW::Mechanise, use WWW::Mechanize or WWW::Scripter.

    Secondly, the link which you've given , that site is created in ASP and believe me it's tough to crawl ASP site, since they use many other different method like EVENTMETHOD, EVENTVALIDATION.............you can do one thing, try to find out links in console and from that you can crawl it. That link will be something like this:
    http://support.dell.com/support/public/menu.aspx?c=us&l=en&s=dhs&cs=19