Mechanize giving errors for no reason

[Anonymous Monk]

So, I'm trying to make a web crawler and I'm getting errors from WWW::Mechanize when they're perfectly valid links. I even tried getting the link without putting it in my recursive crawler and it worked. I'm guessing the website is preventing me from getting their pages after a certain search limit is reached like the Google Search API. Do some websites have a limit as to how much you can crawl through their website?

[space_monk]

Connection could also be rate limited in some way. You'd perhaps do better putting some code up other than have us make random guesses... :-)

[pemungkah]

You might also be getting rate limited because your user agent string says you're WWW::Mechanize. Change it the agent argument in new(). But yes, do check if there's a rate limit, and make sure you're conforming to the site's requirements for access.

[talexb]

Yes, some sites do watch to see that clients aren't hammering their site -- which is reasonable, when you think about it.

We'd be able to help you a great deal more if you were more specific about *which* errors you were seeing.

And .. do you think that Mechanize *randomly* spits out errors? There's almost always a very good reason for errors.