Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re^3: Scrappy user_agent error

by jethro (Monsignor)
on Jan 03, 2012 at 17:51 UTC ( #946103=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Scrappy user_agent error
in thread Scrappy user_agent error

Could it be that you are using a recent scrappy (0.9xxx) but reading the documentation for an older version (like 0.6xxx)? I could find code like "qw/:syntax/" only in older documentation and in example scripts on the web (with a quick google search)


Comment on Re^3: Scrappy user_agent error
Re^4: Scrappy user_agent error
by docster (Novice) on Jan 03, 2012 at 18:45 UTC
    Yes, it is entirely possible. I took the code above from the authors blog post. It is hard to find examples of a working Scrappy script. But as of now I am using: Scrappy (0.94112090).

    And CPANs Module Version: 0.94112090 docs from: http://search.cpan.org/dist/Scrappy/lib/Scrappy.pm#user_agent

    user_agent The user_agent attribute holds the Scrappy::Scraper::UserAgent object which is used to set and manipulate the user-agent header of the scraper.
    my $scraper = Scrappy->new; $scraper->user_agent;
    So in that context, how would I set the user_agent correctly to be firefox using Scrappy 0.94112090? There used to be way. Maybe it was removed. I seem to be missing the entire picture somehow :)

      docster:

      I've not used Scrappy, but perhaps you could check out the tests for examples on how to change the user agent name.

      ...roboticus

      When your only tool is a hammer, all problems look like your thumb.

        Tested this but it still gives an error. Maybe Scrappy is just not the right tool for this job..
        Can't locate object method "user_agent" via package "scraper" (perhaps + you forgot to load "scraper"?) at ./scrappy.pl line 8. #!/opt/local/bin/perl use strict; use warnings; use Scrappy; my $url = 'http://google.com'; my $scraper = Scrappy->new; scraper->user_agent("opera","Macintosh"); $scraper->get("$url"); print $scraper->domain, "\n"; # print www.google.com __END__

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://946103]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (8)
As of 2014-08-31 06:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (294 votes), past polls