Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Re^4: Scrappy user_agent error

by docster (Novice)
on Jan 03, 2012 at 18:45 UTC ( #946111=note: print w/ replies, xml ) Need Help??

in reply to Re^3: Scrappy user_agent error
in thread Scrappy user_agent error

Yes, it is entirely possible. I took the code above from the authors blog post. It is hard to find examples of a working Scrappy script. But as of now I am using: Scrappy (0.94112090).

And CPANs Module Version: 0.94112090 docs from:

user_agent The user_agent attribute holds the Scrappy::Scraper::UserAgent object which is used to set and manipulate the user-agent header of the scraper.
my $scraper = Scrappy->new; $scraper->user_agent;
So in that context, how would I set the user_agent correctly to be firefox using Scrappy 0.94112090? There used to be way. Maybe it was removed. I seem to be missing the entire picture somehow :)

Comment on Re^4: Scrappy user_agent error
Select or Download Code
Replies are listed 'Best First'.
Re^5: Scrappy user_agent error
by jethro (Monsignor) on Jan 04, 2012 at 10:16 UTC
      Tested this but it still gives an error. Maybe Scrappy is just not the right tool for this job..
      Can't locate object method "user_agent" via package "scraper" (perhaps + you forgot to load "scraper"?) at ./ line 8. #!/opt/local/bin/perl use strict; use warnings; use Scrappy; my $url = ''; my $scraper = Scrappy->new; scraper->user_agent("opera","Macintosh"); $scraper->get("$url"); print $scraper->domain, "\n"; # print __END__
Re^5: Scrappy user_agent error
by roboticus (Chancellor) on Jan 04, 2012 at 04:01 UTC


    I've not used Scrappy, but perhaps you could check out the tests for examples on how to change the user agent name.


    When your only tool is a hammer, all problems look like your thumb.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://946111]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (4)
As of 2015-11-29 18:13 GMT
Find Nodes?
    Voting Booth?

    What would be the most significant thing to happen if a rope (or wire) tied the Earth and the Moon together?

    Results (752 votes), past polls