Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
All,
Normally if a site doesn't work with WWW::Mechanize due to JavaScript/Ajax, I reach for WWW::Selenium. This has worked quite well for me in the past (see Using WWW::Selenium To Test Or Automate An Ajax Website). While I am aware of WWW::Scripter, Win32::IE::Mechanize and WWW::Mechanize::Firefox - I have always just reached for one of the two that I have invested the most effort and energy in.

Lately, I have been freelancing and a number of my clients are on linux but still not very computer savvy. Having to start up an Xterm and export a DISPLAY would likely be perceived as clunky software and not win me any repeat business. I considered Running Selenium Headless but that posed its own problems for delivering code to a client. What I needed was a way to make WWW::Mechanize work. I reached for the Firebug Addon to Firefox. The first particular project had 2 hurdles. The first was that clicking on "links" caused the page content to change but the visible URL to stay the same. This was resolved by examining what Firefox was GETing behind the scenes. The next hurdle was that selecting a particular item in a select drop down was sending a JSON request behind the scenes. After much gnashing of teeth, I discovered this little gem

sub post_json { my ($mech, $json, $url) = @_; my $req = HTTP::Request->new(POST => $url); $req->content_type('application/json'); $req->content($json); return $mech->request($req); }

The next project I used Firebug on really had me baffled. The site didn't appear to require Javascript at all and yet I was getting completely different results from WWW::Mechanize and with Firefox. I made sure I was using $mech->agent_alias('Windows Mozilla'); but to no avail. Upon further examining the headers sent between the two, I played a hunch and did the following:

for my $key (keys %ff_header) { $mech->delete_header($key); $mech->add_header($key => $ff_header{$key}); }

It magically started working as expected.

I feel like this is probably old news to most of you and that there are shiny new tools I should be learning. What are they? Do you use certain ones for certain tasks but not others? I realize that some sites will be nearly impossible to automate with WWW::Mechanize without a JavaScript engine and I am fine with that. I am just looking to increase the number of projects I can complete with just mech.

Cheers - L~R


In reply to What Tools Do You Use With WWW::Mechanize by Limbic~Region

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others avoiding work at the Monastery: (9)
    As of 2014-09-22 11:04 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      How do you remember the number of days in each month?











      Results (189 votes), past polls