Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Perl Web Client to Interact with Dynamic Web Pages

by WebDFeat (Initiate)
on Dec 09, 2010 at 19:05 UTC ( #876304=perlquestion: print w/replies, xml ) Need Help??
WebDFeat has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to write a web-client to upload some data (not just a file)automatically to a web-server.

When the web-site is accessed interactively, in a browser, dynamic web-pages are displayed, in response to user input and interaction. The goal is to provide values for fields on these pages, from the Perl client, and to cause these values to be sent to the web-server, as they would from the browser.

I have installed Firebug in my copy of Firefox, in order to understand the DOM structure (?) of the dynamic pages, which is not publicly documented

In my Perl installation, I have installed "WWW:Scripter", with the Javascript plug-in and the SpiderMonkey engine. However, I am unsure how to access the dynamic page content (which I believe should be being created by JavaScript, sent from the server), from within Perl.

Using $response->content, I have only been able to see the original, vestigial HTML structure, prior to its modification by the JavaScript.

I would be delighted to hear from someone with more experience of this to know whether "WWW::Scripter" is a blind alley and, if it is not, to get a example which would help me make progress.

On another, less dynamic site, I did have some success with "WWW:Mechanize", which I used to build up some experience of handling a proxy and using secure http.

Many thanks.

  • Comment on Perl Web Client to Interact with Dynamic Web Pages

Replies are listed 'Best First'.
Re: Perl Web Client to Interact with Dynamic Web Pages
by marto (Bishop) on Dec 09, 2010 at 19:10 UTC
Re: Perl Web Client to Interact with Dynamic Web Pages
by Corion (Pope) on Dec 09, 2010 at 19:12 UTC

    I have no experience with WWW::Scripter, but I assume that you can get at the current DOM by using ->document, and then extracting the elements from there.

    I haven't looked at unifying the API between WWW::Mechanize::Firefox and WWW::Scripter. WWW::Mechanize::Firefox offers ->selector and ->xpath for element extraction, but it requires Firefox to be running.

Re: Perl Web Client to Interact with Dynamic Web Pages
by technojosh (Priest) on Dec 09, 2010 at 21:05 UTC
    WWW::Selenium has already been recommended. Also check the Selenium homepage for details of the firefox plugin available for it. I have found that tool is the way to go when it comes to automating complex web interactions.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://876304]
Approved by marto
[1nickt]: marioroy Yes, I am using it with MCE, as is Discipulus I believe. I was trying to work out how to make a cpanfile that would be smart enough to know which deps to require.

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (8)
As of 2017-10-18 15:29 GMT
Find Nodes?
    Voting Booth?
    My fridge is mostly full of:

    Results (249 votes). Check out past polls.