Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Recommendations on a module for web crawling

by edimusrex (Monk)
on Jul 25, 2016 at 17:59 UTC ( #1168484=perlquestion: print w/replies, xml ) Need Help??

edimusrex has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to crawl a web page where the data I am seeking is generated by javascript. I haven't had to deal with this before but I can imagine I am not the only one who has come across this. Is there a specific module that is preferred in this instance? I've tried using Mechanize::Firefox but the page complains about unsupported browser. I'd prefer to do this in perl but will use other languages if need be.

Thanks for your recommendations

  • Comment on Recommendations on a module for web crawling

Replies are listed 'Best First'.
Re: Recommendations on a module for web crawling
by hippo (Chancellor) on Jul 25, 2016 at 18:31 UTC
    I've tried using Mechanize::Firefox but the page complains about unsupported browser.

    Assuming you mean WWW::Mechanize::Firefox, then yes that would be the first port of call. If the site kicks you out perhaps they don't want any bots at all - check their Ts and Cs and honour them.

    Without any specifics there's little more to suggest.

      Thanks for the response. I am just trying to monitor a site for when it updates its inventory. The javascript must be building the list from a database before displaying it. I will look into WWW::Mechanize::Firefox some more and see if I can't dig a little deeper.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1168484]
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (4)
As of 2021-09-28 05:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?