Re^3: Scrappy user_agent error

[jethro]

Could it be that you are using a recent scrappy (0.9xxx) but reading the documentation for an older version (like 0.6xxx)? I could find code like "qw/:syntax/" only in older documentation and in example scripts on the web (with a quick google search)

[docster]

Yes, it is entirely possible. I took the code above from the authors blog post. It is hard to find examples of a working Scrappy script. But as of now I am using: Scrappy (0.94112090).

And CPANs Module Version: 0.94112090 docs from: http://search.cpan.org/dist/Scrappy/lib/Scrappy.pm#user_agent

user_agent The user_agent attribute holds the Scrappy::Scraper::UserAgent object which is used to set and manipulate the user-agent header of the scraper.

    my  $scraper = Scrappy->new;
        $scraper->user_agent;

So in that context, how would I set the user_agent correctly to be firefox using Scrappy 0.94112090? There used to be way. Maybe it was removed. I seem to be missing the entire picture somehow :)

[jethro]

Look at the source https://metacpan.org/source/AWNCORP/Scrappy-0.94112090/lib/Scrappy/Scraper/UserAgent.pm, first parameter is browser, second parameter the operating system, i.e. you would use something like scraper->user_agent("opera","Macintosh")

PS: As the others have said, if the sites doesn't want robots, you shouldn't just ignore that.

[docster]

Tested this but it still gives an error. Maybe Scrappy is just not the right tool for this job..

Can't locate object method "user_agent" via package "scraper" (perhaps
+ you forgot to load "scraper"?) at ./scrappy.pl line 8.


#!/opt/local/bin/perl
use strict;
use warnings;
use Scrappy;

my $url = 'http://google.com';
my $scraper = Scrappy->new;
scraper->user_agent("opera","Macintosh");
$scraper->get("$url");
print $scraper->domain, "\n"; # print www.google.com

__END__

[roboticus]

docster:

I've not used Scrappy, but perhaps you could check out the tests for examples on how to change the user agent name.

...roboticus

When your only tool is a hammer, all problems look like your thumb.