Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Mechanize and javascript (yes, another)

by bcdeery (Novice)
on Jan 06, 2006 at 15:22 UTC ( #521497=perlquestion: print w/replies, xml ) Need Help??

bcdeery has asked for the wisdom of the Perl Monks concerning the following question:

So I understand Mechanize doen't really understand javascript. But on other posts I see that people can "reverse engineer" it so they can post a form. How? My program has mechanize get the url: "http://www.dhl-usa.com/TransitTimes/USTTimeStart.asp?nav=TransitTimes" I'm trying to automate the retrival of the transit days on the next page (After inputing origin and destination zips). This should be a simple task except for the fact that the "Calculate" button is javascript. I tried going to the page I needed and then just replacing the origin and destination zip codes in the url, but then the table that shows up is wrong-- it just gives the same table no matter how long the shipment would really take. I am very new to the Perl community and appreciate all or any help.
  • Comment on Mechanize and javascript (yes, another)

Replies are listed 'Best First'.
Re: Mechanize and javascript (yes, another)
by marto (Cardinal) on Jan 06, 2006 at 15:29 UTC
    bcdeery,

    When I have had to reverse engineer JavaScript code I use the 'view source' option of my browser (or simply retrieve the file if its an include file) and port the code over to Perl. However, from having looked at this page, all the submitTTime() function (which is called when you click 'Calculate') does is sets a hidden form value and then submit the form. If you use WWW::Mechanize you can set the form value and submit the form.

    The JavaScript line:

    document.frmSvcCalc.hdnAction.value = "Calculate";

    Sets the value of hdnAction to "Calculate".
    To use WWW::Mechanize to do this try:

    $mech->field( "hdnAction", "Calculate" );

    before submitting the form.
    Sadly I can not test this right now (no Perl on desktop machine, no internet access on our development boxes), let me know if you have any problems.

    Hope this helps.

    Martin
      Yes this worked. When it prints URI it gives the original url, but when I print the code to a file, I can verify that it has indeed gone to the correct page. Any idea why that is?

      More importantly, now when I have perl search through the response, it won't find this phrase

      "DHL Ground Service</a> <br> (Letter &#150; 150 Pounds)</div></td> + <td><div class="pL5">Monday,&nbsp;Jan&nbsp;9,&nbsp;2006 &n +bsp;By end of day<img src="http://www.dhl-usa.com/images/blank.gif" w +idth="1" height="1"></div></td> <td><div class="pL5">1"
      but it will find "help you create, print, and pay your shipping" which is only found on the same page.

      I need to extract the last "1" from the string. Any ideas how? Do I have to do a search for "DHL Ground Service" and then somehow search only what is in the response after that for the string connected to the number one that I'm looking for? Sorry this post is funny, I don't know how to disable the HTML tags on my post. Thanks again to anybody who can help!

        Hi bcdeery,

        Looks like you forgot to login before posting your reply.

        "Yes this worked. When it prints URI it gives the original url, but when I print the code to a file, I can verify that it has indeed gone to the correct page. Any idea why that is?"

        It is because the content of the page is dynamically generated. The URL is the same, but the content of the page is different because you submitted the form with the values needed to process your input (inserting values into the form fields and setting "hdnAction" = "Calculate"). As for dealing with the output, I prefer to use HTML::TokeParser or HTML::TokeParser::Simple to parse out the data I want from the HTML tags. I prefer this method to using a regex as you can't always depend on good quality data, and these modules are tried and tested :) Read the module documentation get the drift of how to use them, I wont post an example as there are good examples to work with in the docs.

        Hope this helps.

        Martin
Re: Mechanize and javascript (yes, another)
by BigRare (Pilgrim) on Jan 06, 2006 at 20:42 UTC
Re: Mechanize and javascript (yes, another)
by puff (Beadle) on Jan 06, 2006 at 23:37 UTC
    You might want to take a look at IE::Mechanize.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://521497]
Approved by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (4)
As of 2021-03-05 04:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My favorite kind of desktop background is:











    Results (109 votes). Check out past polls.

    Notices?