http://www.perlmonks.org?node_id=370707


in reply to Remote Login and Form Posting

Whenever you are automating a website, it is of utmost importance to trace what your "real" browser sends and then to try to replicate that with your script. I can recommend the Mozilla HTTP Liveheaders or HTTP::Recorder to sniff the traffic between your browser and the website.

WWW::Mechanize specifically caters towards behaving like a "normal" browser, so you are maybe better off using WWW::Mechanize instead of LWP::UserAgent directly. All methods of LWP::UserAgent are available via WWW::Mechanize as well, so you don't lose anything, but you get magic redirect and referrer handling.

While you are developing your website automation, you will often want to compare what your browser sends and what your script sends, and make the messages as identical as possible, so having a diff utility handy and pumping up the LWP logging might be helpfull too.