Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re^3: How to scrape an HTTPS website that has JavaScript

by tomhukins (Curate)
on Aug 23, 2004 at 17:08 UTC ( #385150=note: print w/replies, xml ) Need Help??


in reply to Re^2: How to scrape an HTTPS website that has JavaScript
in thread How to scrape an HTTPS website that has JavaScript

It was a few months ago, but I recall successfully using HTTP::Recorder to use HTTP::Proxy with JavaScript enabled sites. HTTP::Proxy traps all HTTP requests, regardless of whether they are plain HTML hyperlinks or initiated by JavaScript. Granted, this only works when JavaScript does something simple, but in my experience it usually does. If the JavaScript does anything complex, then you're right: you'll have to either run a JavaScript interpreter within Perl or rewrite the algorithm in Perl.
  • Comment on Re^3: How to scrape an HTTPS website that has JavaScript

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://385150]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (8)
As of 2021-01-27 11:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?