Hello all. I am attempting to write a web crawler in Scrappy. All I can find are posts on how easy it is to do, but not really how to... To me, it is like swimming through mud. I seem to have one issue after another with it. I thought maybe I had some corrupt perl modules so I tested it on a couple of different machines, Mac and Ubuntu Linux. They act the same. In this script it is the user_agent. This is pretty much directly from cpan. What am I missing? ( Posts to stop breathing and die will be ignored. ):
use Scrappy qw/:syntax/;
my $url = 'http://google.com';
my $scraper = Scrappy->new;
print $scraper->domain; # print www.google.com
This script returns:
Can't locate object method "user_agent" via package "random_ua" (perha
+ps you forgot to load "random_ua"?) at ./scrappy.pl line 5.