Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

To Automate through a Dynamic Page like that of Twitter, Using www:mechanize:firefox

by Matt™ (Acolyte)
on Sep 21, 2012 at 13:14 UTC ( #994898=perlquestion: print w/ replies, xml ) Need Help??
Matt™ has asked for the wisdom of the Perl Monks concerning the following question:

To Automate through a dynamic page like that of twitter Using www:mechanize:firefox .

Guten Tag Monks, I need your help to automate through a Dynamic Page like that of twitter. In twitter the initial loading of tweets are about 20 Nos. We need to scroll to the bottom in-order to load another 20 more. So my humble question is, how can i do it via www:mechanize:firefox. It is basically a GET that is happening there but I can't use www:mechanize for this. I am pretty new to www:mechanize:firefox, so pls be kind to answer my question. Thanks in advance.

Bitte Hilfe!, Monks.

Danke.

Comment on To Automate through a Dynamic Page like that of Twitter, Using www:mechanize:firefox
Replies are listed 'Best First'.
Re: To Automate through a Dynamic Page like that of Twitter, Using www:mechanize:firefox
by Corion (Pope) on Sep 21, 2012 at 13:38 UTC

    There are just two steps to do that:

    1. Find out how a human navigates the page
    2. Replicate those steps using WWW::Mechanize::Firefox

    You haven't told us where you have problems, so I suggest you start at step one.

      Thanks Corion. We need to scroll down the page of twitter in-order to get the next set of 20 tweets, Can we replicate the Scroll down process in WWW::Mechanize::Firefox or, is there any alternate way? Thanks in advance.

        There is no direct way with WWW::Mechanize::Firefox. But you can run arbitrary Javascript through it, so you can use that to scroll the page down.

        Personally, I would look at what function gets called to load more information.

        Also, you are certainly aware of the Twitter API and/or Net::Twitter?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://994898]
Approved by herveus
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (12)
As of 2015-07-28 21:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (259 votes), past polls