Hello all. I am attempting to write a web crawler in Scrappy. All I can find are posts on how easy it is to do, but not really how to... To me, it is like swimming through mud. I seem to have one issue after another with it. I thought maybe I had some corrupt perl modules so I tested it on a couple of different machines, Mac and Ubuntu Linux. They act the same. In this script it is the user_agent. This is pretty much directly from cpan. What am I missing? ( Posts to stop breathing and die will be ignored. ):
#!/opt/local/bin/perl
use strict;
use warnings;
use Scrappy qw/:syntax/;
user_agent random_ua;
my $url = 'http://google.com';
my $scraper = Scrappy->new;
$scraper->get("$url");
print $scraper->domain; # print www.google.com
__END__
This script returns:
Can't locate object method "user_agent" via package "random_ua" (perha
+ps you forgot to load "random_ua"?) at ./scrappy.pl line 5.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|