Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re^5: Scrappy user_agent error

by jethro (Monsignor)
on Jan 04, 2012 at 10:16 UTC ( #946216=note: print w/replies, xml ) Need Help??


in reply to Re^4: Scrappy user_agent error
in thread Scrappy user_agent error

Look at the source https://metacpan.org/source/AWNCORP/Scrappy-0.94112090/lib/Scrappy/Scraper/UserAgent.pm, first parameter is browser, second parameter the operating system, i.e. you would use something like scraper->user_agent("opera","Macintosh")

PS: As the others have said, if the sites doesn't want robots, you shouldn't just ignore that.

Replies are listed 'Best First'.
Re^6: Scrappy user_agent error
by docster (Novice) on Jan 06, 2012 at 16:51 UTC
    Tested this but it still gives an error. Maybe Scrappy is just not the right tool for this job..
    Can't locate object method "user_agent" via package "scraper" (perhaps + you forgot to load "scraper"?) at ./scrappy.pl line 8. #!/opt/local/bin/perl use strict; use warnings; use Scrappy; my $url = 'http://google.com'; my $scraper = Scrappy->new; scraper->user_agent("opera","Macintosh"); $scraper->get("$url"); print $scraper->domain, "\n"; # print www.google.com __END__

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://946216]
help
Chatterbox?
[zentara]: LanX One day, I will see a great interest in Tk. It's small, self-contained, needs very few external libraries, AND makes it easy to make high-contrast displays. :-)
[zentara]: Corion in this cased they left the www. out
[zentara]: /d// :-)
[LanX]: i like tk, just joking. but what about modern styles?
LanX has to run. .. o/
[zentara]: modern styles are secondary to display readability. Tk is automatically good at the clunky industrial theme. :-)
[ww]: Amen!

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (8)
As of 2017-05-26 12:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?