LWP::UserAgent gets 403 when other browser does not

[pileofrogs]

Greetings Monks!

I'm writing a link checker and I've noticed that I get '403' errors for some pages with LWP::UserAgent, but if I try the same URLs from firefox, I get the content, no problem. I'm doing $ua->get($uri) and $ua->head($uri) requests.

My first thought was that LWP::UserAgent was doing some kind of robots.txt handling without telling me, but there's LWP::RobotUA for that, so I assume that's not the case. (Is my assumption wrong?)

Does anyone know a reason why certain web pages would return a status 403 to LWP::UserAgent, but work fine with a normal web browser?

Thanks!

--Pileofrogs

[ikegami]

Most likely because required cookies are missing or because of some robot detection being done by the server.

Match your request more closely with what Firefox sends, possibly going as far as using Mozilla to do the request.

[pemungkah]

WWW::Mechanize supports user agent aliases right out of the box. I too lean toward robot detection via signature, and suggest either Firefox or IE as your most-likely-to-be-accepted alternatives.

[ikegami]

So does LWP::UserAgent. The only difference is that WWW::Mechanize provides these aliases:

'Windows IE 6'    => 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.
+1)',
'Windows Mozilla' => 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; 
+rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6',
'Mac Safari'      => 'Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) 
+AppleWebKit/85 (KHTML, like Gecko) Safari/85',
'Mozilla'         => 'Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; 
+en-US; rv:1.4a) Gecko/20030401',
'Linux Mozilla'   => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) 
+Gecko/20030624',
'Linux Konqueror' => 'Mozilla/5.0 (compatible; Konqueror/3; Linux)',

[pileofrogs]

Thanks all!

I decided that I would try multiple times with different settings before declaring a link bad. Some servers don't do HEAD requests, others need a useragent and some want a referrer.

This makes the whole thing slower, but that doesn't matter when it runs automatically at night.

Thanks again!

--Pileofrogs