Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Re: 2 questions for Corion regarding href with Mechanize::Firefox

by LanX (Bishop)
on Jul 15, 2013 at 20:53 UTC ( #1044457=note: print w/replies, xml ) Need Help??

in reply to 2 questions for Corion regarding href with Mechanize::Firefox

You have to decide when to poll the links.

The module has only limited means to tell when a page is "ready" (it's meant to mechanize dynamic pages! remember?)

That's why you have to check what "ready" means for you (e.g. testing if a special DOM-element already appeared) and synchronize your actions.

Cheers Rolf

( addicted to the Perl Programming Language)

  • Comment on Re: 2 questions for Corion regarding href with Mechanize::Firefox

Replies are listed 'Best First'.
Re^2: 2 questions for Corion regarding href with Mechanize::Firefox
by help_3452 (Initiate) on Jul 15, 2013 at 22:46 UTC

    Error Code :
    No link found matching '//a(@href = ";jsessionid=0001xxxxxxxxxxxxxxxxxxxxxxx:xxxxxxxxxxx?" or @src=";jsessionid=0001xxxxxxxxxxxxxxxxxxxxxxx:xxxxxxxxxxx?")'

    Actual HTML:

    <a id="register" tabindex="106" href="/CAA/jsp/register_customer.jsp;j +sessionid=0001xxxxxxxxxxxxxxxxxxxxxxx:xxxxxxxxxxx?targeturl=https%3A% +html"> Register</a>

    Found @ :;jsessionid=0001xxxxxxxxxxxxxxxxxxxxxxx:xxxxxxxxxxx?

    (Probably best found by navigating to and then click login.)
    The register link on the login page is what I'm trying demo $mech->follow_link

    Using Code:
    @arr = $mech->find_all_links; $link_obj = @arr[4];

    Further on I have
    eval {$mech->follow_link( url => $link_obj->url, tag => $link_obj-> +tag) };

    These two methods don't work consistently.

    To respond to all of you in turn.

    moritz: Ok. I do need to have the ability to navigate/edit javascript. That part I have covered.
    micheal: Yes I've looked at the documentation. Yes having now gone through pretty much all the code for www::mechanize::firefox it is clear to me. The code i've quoted is reasonable and logical. The documentation only needs one extra line to point out that follow_link will not necessarily work with the links found by find_all_links. Or if I understand the code follow_link will search the html while find_all_links will refer to the DOM, which is not documented.
    anonymous monk: This is really aggressive! I'm just asking a question. 1. mechanize::firefox doesn't exit. ANS you are pedantic. 2. terms of service. I don't want to scrape either website. I fact i don't want to scrape any website, certainly not any that i don't have permission to. Both websites cited here are only mentioned so i can show the problem. 3. API db access. That is just aggressive and unhelpful. 4. The documentation could be a little clearer on the follow_links section. 5. submit a patch. sure no problem. This is my fifth day learning perl, so once i have the skill I would be very happy to help improve the code. 6. say something meaningful. www::mechanize::firefox is a super piece of work, it is very slick. I'm only quibbling about a small element, because I would like others to use this cool software and would not like them to get confused as i did.

    bottom line is find_all_links returns objects which are then not usable with follow_link. This is not what I would expect.
    Rolf: Cheers but no that is not the problem. It is not a question of the data being unavailable. I can access the data.

      Thanks for posting actual code and data.

      I've never used ->find_all_links in conjunction with ->follow_link. As ->find_all_links needs to return absolute links, and as there is no easy way for ->follow_link to determine URL equivalence for arbitrary href attributes, the problem is basically unsolvable that way.

      As a workaround, I would look at ->find_link_dom , which returns DOM objects instead of converting things to strings.

        Thank you so much for responding. Your code is dense and I wasn't sure I understood it correctly.

        Really appreciate you taking the time to respond.

        Should I come up with a good solution over the next while I will share it.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1044457]
and God said, "Let Newton be!"...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (3)
As of 2018-05-25 05:40 GMT
Find Nodes?
    Voting Booth?