Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

How do I get WWW::Mechanize::PhantomJS to wait until the JavaScript on the web page has finished executing?

by Darth Ed (Beadle)
on Aug 20, 2021 at 20:36 UTC ( [id://11135995] : perlquestion . print w/replies, xml ) Need Help??

Darth Ed has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to use WWW::Mechanize::PhantomJS with a JavaScript-heavy web application.

The decoded_content that WWW::Mechanize::PhantomJS returns shows the web page in an unfinished state.

How do I get WWW::Mechanize::PhantomJS either to wait until the JavaScript on the web page has finished executing or to just wait a couple seconds after the page loads when I $mech->get?

Thanks!

  • Comment on How do I get WWW::Mechanize::PhantomJS to wait until the JavaScript on the web page has finished executing?
  • Select or Download Code

Replies are listed 'Best First'.
Re: How do I get WWW::Mechanize::PhantomJS to wait until the JavaScript on the web page has finished executing?
by LanX (Saint) on Aug 21, 2021 at 01:35 UTC
    Browsers have no reliable event saying "JS has finished executing", because there might be a timer constantly executing functions or callbacks triggered by returning AJAX requests ... and so on.

    Note also that JS can run in parallel.

    So you have to tell Mechanize to wait for an DOM-element to appear which represent the state you consider "finished".

    When working with WWW::Mechanize::Firefox many years ago I used this Wait-until-an-element-appears approach.

    You have to see how this best translates to PhantomJS and your use case.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

      So you have to tell Mechanize to wait for an DOM-element to appear which represent the state you consider "finished".

      Sure, I get that, but I haven't been able to figure out how to do that with WWW::Mechanize::PhantomJS. I don't see an is_visible method listed in the POD.... I'd be fine with just setting a timer for 3 seconds, but, again, I don't see how to do that either.

        Using one of the selectors in WWW::Mechanize::PhantomJS to find a "finished" characteristic (DOM element or attribute or combination or ...) should return an empty result unless it exists. (See all EXTRACTION-METHODS)

        And even if this didn't work, you could ->eval your own JS-code which does those checks in a loop for you and only returns after success.

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        Wikisyntax for the Monastery

        Maybe just use sleep to wait for three seconds?