Re: www::mechanize not behaving as browser?

[Anonymous Monk]

WWW::Mechanize is behaving as a browser.

It is the stupid servers/cgi programs which is making assumptions, basically expecting a specific brand of browser and exploding when it gets something different.

So if you're going to fool these stupid programs, you have to pretend you are that specific brand of browser.

The way you do this is by using a regular browser to navigate the website successfully while you record the HTTP conversation with HTTP::Recorder/WireShark/Ethereal/LiveHTTPHeaders... and then you configure WWW::Mechanize to send similar headers....

[jonnyfolk]

Hi, thanks for your comment - I am taking a look at Wireshark now.

I have tried several different browsers include IE & Firefox on Windows and Safari, Firefox and even Camino on Mac, and all have achieved successful search results. It doesn't seem too fussy about the particular browser, but there's obviously something different in my Mechanize browser which is sending things awry.

[Anonymous Monk]

If it isn't filtering by the user-agent string, its probably setting a cookie via img (which mechanize doesn't load), or some form value via script... so it could be as simple as fetching an image, or set some extra form values normally set via script ... it all shows up on the wire :)