laziness, impatience, and hubris | |
PerlMonks |
What Tools Do You Use With WWW::Mechanizeby Limbic~Region (Chancellor) |
on Oct 03, 2011 at 14:13 UTC ( [id://929348]=perlquestion: print w/replies, xml ) | Need Help?? |
Limbic~Region has asked for the wisdom of the Perl Monks concerning the following question:
All,
Normally if a site doesn't work with WWW::Mechanize due to JavaScript/Ajax, I reach for WWW::Selenium. This has worked quite well for me in the past (see Using WWW::Selenium To Test Or Automate An Ajax Website). While I am aware of WWW::Scripter, Win32::IE::Mechanize and WWW::Mechanize::Firefox - I have always just reached for one of the two that I have invested the most effort and energy in. Lately, I have been freelancing and a number of my clients are on linux but still not very computer savvy. Having to start up an Xterm and export a DISPLAY would likely be perceived as clunky software and not win me any repeat business. I considered Running Selenium Headless but that posed its own problems for delivering code to a client. What I needed was a way to make WWW::Mechanize work. I reached for the Firebug Addon to Firefox. The first particular project had 2 hurdles. The first was that clicking on "links" caused the page content to change but the visible URL to stay the same. This was resolved by examining what Firefox was GETing behind the scenes. The next hurdle was that selecting a particular item in a select drop down was sending a JSON request behind the scenes. After much gnashing of teeth, I discovered this little gem
The next project I used Firebug on really had me baffled. The site didn't appear to require Javascript at all and yet I was getting completely different results from WWW::Mechanize and with Firefox. I made sure I was using $mech->agent_alias('Windows Mozilla'); but to no avail. Upon further examining the headers sent between the two, I played a hunch and did the following:
It magically started working as expected. I feel like this is probably old news to most of you and that there are shiny new tools I should be learning. What are they? Do you use certain ones for certain tasks but not others? I realize that some sites will be nearly impossible to automate with WWW::Mechanize without a JavaScript engine and I am fine with that. I am just looking to increase the number of projects I can complete with just mech. Cheers - L~R
Back to
Seekers of Perl Wisdom
|
|