Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

web scraping with javascript support?

by gulden (Monk)
on Aug 03, 2010 at 16:34 UTC ( #852699=perlquestion: print w/replies, xml ) Need Help??
gulden has asked for the wisdom of the Perl Monks concerning the following question:

I need to scrape webpages, where parts of the HTML page is obtained through the execution of Javascript function. For instance i need to scrape the inner table of this site

There is a library to do this? How can I get a workaround around to do it, once I want to do this for several pages/sites?

WWW::Mechanize is not the solution: «That's because WWW::Mechanize doesn't operate on the JavaScript. It only understands the HTML parts of the page.»

$ wget also doesn't suppport it

Any workaround/solution?

«A contentious debate is always associated with a lack of valid arguments.»

Replies are listed 'Best First'.
Re: web scrapping with javascript support?
by runrig (Abbot) on Aug 03, 2010 at 16:42 UTC
Re: web scrapping with javascript support?
by mojotoad (Monsignor) on Aug 03, 2010 at 20:18 UTC
Re: web scrapping with javascript support?
by merlyn (Sage) on Aug 03, 2010 at 18:33 UTC
    I've often wanted to "scrap" the web. Start over. Try something new. :)

    I presume you mean "scrape", as in "web scraping".

    -- Randal L. Schwartz, Perl hacker

    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Re: web scrapping with javascript support?
by superfrink (Curate) on Aug 03, 2010 at 20:01 UTC

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://852699]
Approved by Old_Gray_Bear
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (8)
As of 2016-10-27 20:08 GMT
Find Nodes?
    Voting Booth?
    How many different varieties (color, size, etc) of socks do you have in your sock drawer?

    Results (370 votes). Check out past polls.