in reply to Re: Scrappy user_agent error
in thread Scrappy user_agent error
Some sites I visit automatically block robots. I was under the impression that this would change the default "Browser" id... Am I wrong?
From CPAN: The user_agent attribute holds the Scrappy::Scraper::UserAg +ent object which is used to set and manipulate the user-agent header +of the scraper. use Scrappy qw/:syntax/; user_agent random_ua; or user_agent random_ua 'firefox'; # firefox only user_agent random_ua 'firefox', 'linux'; # firefox on linux only
|
---|
Replies are listed 'Best First'. | |
---|---|
Re^3: Scrappy user_agent error
by jethro (Monsignor) on Jan 03, 2012 at 17:51 UTC | |
by docster (Novice) on Jan 03, 2012 at 18:45 UTC | |
by jethro (Monsignor) on Jan 04, 2012 at 10:16 UTC | |
by docster (Novice) on Jan 06, 2012 at 16:51 UTC | |
by roboticus (Chancellor) on Jan 04, 2012 at 04:01 UTC | |
Re^3: Scrappy user_agent error
by GrandFather (Saint) on Jan 03, 2012 at 23:43 UTC |
In Section
Seekers of Perl Wisdom