ait has asked for the wisdom of the Perl Monks concerning the following question:


I am trying to learn WWW::Scripter to scrape some JS intensive pages. I tried evaling the JQuery.js from a page received with $mech->get. Since that didn't work I tried a simple example and this doesn't work either. Maybe JQuery needs a loaded DOM for this to work?

#!/usr/bin/env perl use warnings; use strict; use HTTP::Cookies; use WWW::Scripter; my $cookie_jar = HTTP::Cookies->new(); my $mech = WWW::Scripter->new( agent => 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident +/5.0', cookie_jar => $cookie_jar, ); $mech->get(" +min.js"); my $jquery = $mech->content; $mech->use_plugin( JavaScript => engine => 'SpiderMonkey', ); $mech->eval($jquery);

I get "c has no properties at WWW::Scripter::Plugin::JavaScript::SpiderMonkey line 165"

Any help greatly appreciated...

Update 20120306

To further expand on the comment below by tobyink, the fact that it can't parse JQuery is by no means a limitation of WWW::Scripter, it just means that I was trying to use a great tool for the wrong job.

In my particular case I need a full-blown rendering engine so there is not getting around the use of something like Gecko, whether it's using Xvfb, Crowbar, offscreen or something along those lines. I would definitively try to keep everything within WWW::Mechanize, after all, the final scraping is very effective with the tools provided in that namespace, so probably WWW::Mechanize::Firefox and Xvfb is the way to go. Expect another update when I get one of these working...