perlquestion
ait
<p>
Hello,
</p>
<p>
I am trying to learn WWW::Scripter to scrape some JS intensive pages. I tried evaling the JQuery.js from a page received with $mech->get. Since that didn't work I tried a simple example and this doesn't work either. Maybe JQuery needs a loaded DOM for this to work?
</p>
<code>
#!/usr/bin/env perl
use warnings;
use strict;
use HTTP::Cookies;
use WWW::Scripter;
my $cookie_jar = HTTP::Cookies->new();
my $mech = WWW::Scripter->new(
agent => 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0',
cookie_jar => $cookie_jar,
);
$mech->get("https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js");
my $jquery = $mech->content;
$mech->use_plugin(
JavaScript =>
engine => 'SpiderMonkey',
);
$mech->eval($jquery);
</code>
<p>
I get "c has no properties at WWW::Scripter::Plugin::JavaScript::SpiderMonkey line 165"
</p>
<p>
Any help greatly appreciated...
</p>
<h3>Update 20120306</h3>
<p>
To further expand on the comment below by [tobyink], the fact that it can't parse JQuery is by no means a limitation of WWW::Scripter, it just means that I was trying to use a great tool for the wrong job.
</p>
<p>
In my particular case I need a full-blown rendering engine so there is not getting around the use of something like Gecko, whether it's using [http://www.xfree86.org/4.0.1/Xvfb.1.html|Xvfb], [http://simile.mit.edu/wiki/Crowbar|Crowbar], [http://hg.mozilla.org/incubator/offscreen|offscreen] or something along those lines. I would definitively try to keep everything within WWW::Mechanize, after all, the final scraping is very effective with the tools provided in that namespace, so probably WWW::Mechanize::Firefox and Xvfb is the way to go. Expect another update when I get one of these working...
</p>