Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re^5: Scrappy user_agent error

by jethro (Monsignor)
on Jan 04, 2012 at 10:16 UTC ( #946216=note: print w/ replies, xml ) Need Help??


in reply to Re^4: Scrappy user_agent error
in thread Scrappy user_agent error

Look at the source https://metacpan.org/source/AWNCORP/Scrappy-0.94112090/lib/Scrappy/Scraper/UserAgent.pm, first parameter is browser, second parameter the operating system, i.e. you would use something like scraper->user_agent("opera","Macintosh")

PS: As the others have said, if the sites doesn't want robots, you shouldn't just ignore that.


Comment on Re^5: Scrappy user_agent error
Download Code
Replies are listed 'Best First'.
Re^6: Scrappy user_agent error
by docster (Novice) on Jan 06, 2012 at 16:51 UTC
    Tested this but it still gives an error. Maybe Scrappy is just not the right tool for this job..
    Can't locate object method "user_agent" via package "scraper" (perhaps + you forgot to load "scraper"?) at ./scrappy.pl line 8. #!/opt/local/bin/perl use strict; use warnings; use Scrappy; my $url = 'http://google.com'; my $scraper = Scrappy->new; scraper->user_agent("opera","Macintosh"); $scraper->get("$url"); print $scraper->domain, "\n"; # print www.google.com __END__

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://946216]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (12)
As of 2015-07-30 12:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (271 votes), past polls