Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer

web scraping with javascript support?

by gulden (Monk)
on Aug 03, 2010 at 16:34 UTC ( #852699=perlquestion: print w/replies, xml ) Need Help??
gulden has asked for the wisdom of the Perl Monks concerning the following question:

I need to scrape webpages, where parts of the HTML page is obtained through the execution of Javascript function. For instance i need to scrape the inner table of this site

There is a library to do this? How can I get a workaround around to do it, once I want to do this for several pages/sites?

WWW::Mechanize is not the solution: That's because WWW::Mechanize doesn't operate on the JavaScript. It only understands the HTML parts of the page.

$ wget also doesn't suppport it

Any workaround/solution?

A contentious debate is always associated with a lack of valid arguments.

Replies are listed 'Best First'.
Re: web scrapping with javascript support?
by runrig (Abbot) on Aug 03, 2010 at 16:42 UTC
Re: web scrapping with javascript support?
by mojotoad (Monsignor) on Aug 03, 2010 at 20:18 UTC
Re: web scrapping with javascript support?
by merlyn (Sage) on Aug 03, 2010 at 18:33 UTC
    I've often wanted to "scrap" the web. Start over. Try something new. :)

    I presume you mean "scrape", as in "web scraping".

    -- Randal L. Schwartz, Perl hacker

    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Re: web scrapping with javascript support?
by superfrink (Curate) on Aug 03, 2010 at 20:01 UTC

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://852699]
Approved by Old_Gray_Bear
[marioroy]: choroba++, Discipulus++. It depends on the type of module. Data-type "only" modules are likely multi-process safe, re: Hash::Ordered, Tie::IxHash.
[marioroy]: ... when shared via MCE::Share-> share(...)

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (5)
As of 2017-09-22 10:11 GMT
Find Nodes?
    Voting Booth?
    During the recent solar eclipse, I:

    Results (260 votes). Check out past polls.