Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Scraping Javascript page using perl

by Bpl (Beadle)
on Nov 21, 2020 at 14:39 UTC ( #11123974=perlquestion: print w/replies, xml ) Need Help??

Bpl has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monkers, Actually I am trying to scrape the Javascript code of an Italian website, I am able to scrape the normal HTML code through the classical
$lwp->get( $site );

But I cannot scrape the DOM code created by Javascript (or Ajax), any helps??

I have already tried with the "get" function of WWW::Mechanize::PhantomJS but it doesn't work.

the site is: http://operedigitali.lincei.it/rendicontiFMN/cliccami.htm under the voice of "volumi"

Thanks in advance.

Replies are listed 'Best First'.
Re: Scraping Javascript page using perl
by Corion (Pope) on Nov 21, 2020 at 14:55 UTC
      Hi Corion, the problem with WWW::Mechanize::PhantomJS is that its 'get' function return only the classical HTML code (the page source), not the loaded page with JS.

      Regards, Edoardo Mantovani, 2020

Re: Scraping Javascript page using perl
by markong (Monk) on Nov 21, 2020 at 16:50 UTC

    Hi Edoardo, a quick reply with some pointers in directions worth exploring.

    First things first: if I look at the source of that page, I don't see any JS. I see pretty basic HTML. What I see is also a bunch of frames, and you should look at the source of the pages loaded in those frames if you already get what you're looking for (sorry have no time to do this for you, also you don't specify what you're after).

    If you're also interested in easily executing/parsing some JS (ECMA Script v3) code, I would give something like JE a try!
    Why? Because I see in the source of the page this:

    <meta name="GENERATOR" content="Microsoft FrontPage 3.0">
    and I bet the JS wouldn't be an Angular module, if any!

    Saluti

      Hi, I am currently fixing this using

      https://metacpan.org/pod/WWW::Mechanize::Frames

      The idea is to download every pdf file from the site

      Grazie per L'interessamento.

      Edoardo

Re: Scraping Javascript page using perl
by bliako (Prior) on Nov 21, 2020 at 17:23 UTC

    Quick fix: Open Firefox's Developer Tools, go to the Network tab and observe all the transactions which happen during loading. One of them is requesting the HTML for the actual contents of that page, the 1892 theses (alas only mathematics is immortal), something like this:

    http://operedigitali.lincei.it/rendicontiFMN/rol/visart.php?lang=it&type=mat&serie=5&anno=1892&volume=1

    The longer way which is the "proper" way IMO is to do what Corion suggested and use WWW::Mechanize::Chrome (I am not acquainted with WWW::Mechanize::PhantomJS). This instructs google chrome browser to get the web-page and then asks it to provide Perl with the DOM. Then it's straight forward with XPath selectors to reach and suck out the desired div's contents.

    bw, bliako

      Hi Bliako, nice to see you again, Many thanks for the help!

      The project is growing, you'll have more info next days.

      Still thanks for everything!

        Glad you never give up! All the best

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11123974]
Approved by marto
Front-paged by toolic
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (2)
As of 2020-12-03 08:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How often do you use taint mode?





    Results (53 votes). Check out past polls.

    Notices?