Re: Re: Using URI::URL

This is an archived low-energy page for bots and other anonmyous visitors. Please sign up if you are a human and want to interact.

does www::mechanize follow the rules set by lwp::robotua? I know that before ifi tried to "get" something that a robots.txt file didnt allow me too, my get came up empty (a good thing). Does www:mechanize allow me to do the same thing? I have been to CPAN and looked at it, but didnt see anything about obeying rules. Thanks

Comment on Re: Re: Using URI::URL

Replies are listed 'Best First'.
WWW::Mechanize::Polite ? by jeffa (Bishop) on Feb 21, 2004 at 21:58 UTC
Kudos to you for wanting polite bots. The problem with getting LWP::RobotUA to play nice with WWW::Mechanize is that they both are subclasses of LWP::UserAgent. By itself, WWW::Mechanize does not consult the `/robots.txt` file, but you can instead use WWW::RobotRules. Here is a working example that tries to grab two files from my server: Read more... (902 Bytes) There might be a better way though ... ahh, how about "WWW::Mechanize::Polite"? Read more... (1456 Bytes) And if i didn't just reinvent a wheel, you might be seeing this on the CPAN. ;) jeffa L-LL-L--L-LL-L--L-LL-L-- -R--R-RR-R--R-RR-R--R-RR B--B--B--B--B--B--B--B-- H---H---H---H---H---H--- (the triplet paradiddle with high-hat)	[reply] [d/l] [select]
Re: WWW::Mechanize::Polite ? by mkurtis (Scribe) on Feb 21, 2004 at 22:14 UTC
thanks jeffa, you rock!	[reply]

• hippo

‥ epoptai's answer Re: how do I set a cookie and redirect was blessed by hippo!

• erzuuli

‥ Anonymous Monks are no longer allowed to use Super Search, due to an excessive use of this resource by robots.


Don't ask to ask, just ask
	PerlMonks