How do I perform web automation with sites that use Javascript?

[whakka]

I need to automate a script for crawling web pages, but these pages use Javascript/AJAX for form processing and the like. LWP and WWW::Mechanize don't handle this case well. What can I do?

Originally posted as a Categorized Question.

[jettero]

JavaScript::SpiderMonkey seems to be in use by scripts built as recently as Net::Plurk::Dumper. Something to add to the list in any case.

[jdporter]

Here are some modules which give you a way around the issue:

• WWW::Scripter
• WWW::Selenium
• Win32::IE::Mechanize
• Mozilla::Mechanize

Other things to try:

• Disable Javascript in your browser and see if the site still functions as you want. If so, then you don't actually have a problem :)
• Figure out what the scripts are doing on the wire, and re-implement those transactions in your own program. The Firefox add-on Live HTTP Headers is well suited for this.

This info provided by the OP.

[planetscape]

Don't forget Limbic~Region's excellent Tutorial, Using WWW::Selenium To Test Or Automate An Ajax Website.

[ninuzzo]

A recent addition of mine is WWW::HtmlUnit::Spidey. This module uses the Java library HtmlUnit which is a headless browser with pretty good JavaScript support. Do not worry, you won't have to write any Java code :D

It is good for massive web scraping where screen scraping does not scale and may be unstable.

There is a tutorial here that scrapes some data obtained from a form not working without JavaScript support.

Btw I am just a Perl beginner. Any Perl guru interested in co-developing Spidey?

[Anonymous Monk]

If you wish to recruit, try perl news section, much more visible

[jdporter]

Re: How do I perform web automation with sites that use Javascript?

Originally posted as a Categorized Answer.