|
|
|
| Syntactic Confectionery Delight | |
| PerlMonks |
Re^2: Test::WWW::Mechanize page_links_ok fails on wikipedia entry external linksby mandog (Curate) |
| on Feb 05, 2009 at 21:38 UTC ( [id://741704]=note: print w/replies, xml ) | Need Help?? |
|
Yep, robots.txt / user-agent exclusion is the problem $mech->agent_alias( 'Windows IE 6' ); works with wikipedia but for some reason not gnu.org $mech->agent_alias('Linux Mozilla'); works for both. I guess if wikipedia doesn't want mech scraping, I won't do it. Thanks for your help planetscape,
In Section
Seekers of Perl Wisdom
|
|
||||||||||||||||||||||