Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Re^3: Scrappy user_agent error

by jethro (Monsignor)
on Jan 03, 2012 at 17:51 UTC ( #946103=note: print w/replies, xml ) Need Help??

in reply to Re^2: Scrappy user_agent error
in thread Scrappy user_agent error

Could it be that you are using a recent scrappy (0.9xxx) but reading the documentation for an older version (like 0.6xxx)? I could find code like "qw/:syntax/" only in older documentation and in example scripts on the web (with a quick google search)

Replies are listed 'Best First'.
Re^4: Scrappy user_agent error
by docster (Novice) on Jan 03, 2012 at 18:45 UTC
    Yes, it is entirely possible. I took the code above from the authors blog post. It is hard to find examples of a working Scrappy script. But as of now I am using: Scrappy (0.94112090).

    And CPANs Module Version: 0.94112090 docs from:

    user_agent The user_agent attribute holds the Scrappy::Scraper::UserAgent object which is used to set and manipulate the user-agent header of the scraper.
    my $scraper = Scrappy->new; $scraper->user_agent;
    So in that context, how would I set the user_agent correctly to be firefox using Scrappy 0.94112090? There used to be way. Maybe it was removed. I seem to be missing the entire picture somehow :)
        Tested this but it still gives an error. Maybe Scrappy is just not the right tool for this job..
        Can't locate object method "user_agent" via package "scraper" (perhaps + you forgot to load "scraper"?) at ./ line 8. #!/opt/local/bin/perl use strict; use warnings; use Scrappy; my $url = ''; my $scraper = Scrappy->new; scraper->user_agent("opera","Macintosh"); $scraper->get("$url"); print $scraper->domain, "\n"; # print __END__


      I've not used Scrappy, but perhaps you could check out the tests for examples on how to change the user agent name.


      When your only tool is a hammer, all problems look like your thumb.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://946103]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (5)
As of 2017-04-25 12:33 GMT
Find Nodes?
    Voting Booth?
    I'm a fool:

    Results (451 votes). Check out past polls.