Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Using WWW::Selenium To Test Or Automate An Ajax Website

by Limbic~Region (Chancellor)
on Oct 28, 2008 at 14:49 UTC ( [id://720018]=perltutorial: print w/replies, xml ) Need Help??

All,
If you have ever had needed to automate interaction with a website that relied heavily on JavaScript and Ajax, then you know there are not a lot of perl tools to help you. You can't reach for your trusty WWW::Mechanize. Sure there is WWW::Mechanize::Plugin::JavaScript but it is still in the infantile stages of development. Perhaps you were hoping Win32::IE::Mechanize or Mozilla::Mechanize were the solution. They might be for you, but the limitations and caveats rendered them unfeasible for my project.

The solution for me was to use Selenium. It supports a variety of browsers, platforms, and programming languages - including perl. Selenium has a number of components but the two discussed in this tutorial are the Selenium IDE and Selenium RC. The IDE is a FireFox add-on that allows you to "record" the interaction of a website and generate working test scripts in a variety of languages. The Remote Control (RC) component provides an API controlling Selenium from your favorite language.

Getting Started

I had some problems getting set up so here are some things that you may need to know. I also posted this at the Selenium RC user forum here.

I could not get FF to work using any of the default options Selenium RC provides (*firefox and *chrome). I ended up having to start firefox with -profilemanager and creating a custom profile. This means going in and setting the proxy settings to Selenium's default port (4444) as well trusting the CyberVillian's CA for https support (more on that later). The end result looked like:

"*custom C:/path/to/ff/firefox.exe -no-remote -profile C:/selenium/fir +efox/custom_profile"

I also ran into trouble with my software based firewall for Windows (Norton Internet Security). It allowed Java to open a listening port on 4444 but it would not allow any outgoing connections. Since you can configure different applications with different settings - this is something to watch out for.

In order for Selenium to allow interaction with SSL sites, you need to trust the Cybervillians CA certificate. This is all supposed to be hidden under the covers if you use one of the default browser options like *chrome but I was stuck doing everything myself with *custom. Keep in mind that you want to ONLY set up this trust in the custom profile for obvious security reasons. Additionally, the last stable release of Selenium RC has an expired certificate. You will need to get a current one from one of the nightly builds. Lastly, the Selenium core can be started with -trustAllSSLCertificates which may be required for sites that you want to automate that do not have valid certs themselves.

Selenium gives extremely unfriendly error messages ("Permission denied to get property Window.seleniumMarker") if you attempt to access a site with a different base domain then the one you passed in in the constructor. If you log on to https://example.com and it silently redirects you to https://www.example.com you will get that weird error message if you follow a link on the page. Be sure to start a new browser if you need to go to a site with a different base domain name.

Finally, the output of the IDE is a test script where everything ends in _ok. If you are automating and not testing, you will need to drop all of the _oks as well as remember to call $sel->start before any automation.

How does Selenium make working with Ajax easy?

The Selenium IDE will follow any action such as open() or click() that results in a new page to load with $sel->wait_for_page_to_load(30000); which means to wait 30 seconds for the page to load. This may return before the page is actually finished if things are happening via Ajax. Fortunately you can replace the call to $sel->wait_for_page_to_load(); with a number of other methods.

It is folly to try and use arbitrary pauses such as $sel->pause(5000) or sleep 5. You really should use one of the wait_for alternatives. These methods all follow the same naming scheme such as $sel->wait_for_pop_up($window_id, $timeout) and $sel->wait_for_element_present($locator, $timeout); They can be wrapped in an eval block. This is almost enough, but there are still circumstances when even wait_for will fail you (more on that later).

If you chose not to use the FF IDE to pre-record your session, you may find it difficult trying to figure out what Selenium wants for $locator when waiting for an element. There are a number of different ways to specify a locator (and even an extension to add your own), but xpath seems to be the preferred way. When you are using the IDE, you can right click over just about anything (button, form element, text) and get some additional options such as "assert_text_present" or "assert_element_present". You can then copy/paste the xpath into $locator without having to figure anything else out.

When are the wait_for methods not enough?

I had tried the wait_for methods and they didn't seem to work. I went to arbitrary pauses which intermittently worked. I googled. I looked under the covers. Nothing. It was frustrating since I could see the element I was waiting for plain as day but clicking on it didn't behave the same way as when I interacted with the site myself. The problem turned out to be that the element was loaded before the JS function behind it was. Unfortunately, there is no $sel->wait_for_JS_function($func_name, $timeout); I subclassed and added one.

The secret is that Selenium provides a very powerful method called $sel->get_eval(); which will allow you to execute arbitrary JS and return the results. My subclass looked a bit like this:

sub wait_for_JS_function { my ($self, $func, $timeout) = @_; for (1 .. $timeout / 100) { my $res = $self->get_eval("typeof( window.$func )"); return $res if $res ne 'null'; $self->pause(100); } croak "Timed out waiting for JS func: '$func'"; }

Working With Pop-Ups

Interacting with JS pop-ups is easy ($sel->select_window()). Unfortunately, grabbing windows that were not opened by window.open() but as a result of clicking on a link with a target of _blank is less than straight forward. I created the following subclass method but I will not stand by it not having any bugs:
# Assumes only 1 currently opened window with target _blank sub select_target_blank_window { my ($self, $timeout) = @_; my $window_name; for (1 .. $timeout / 100) { ($window_name) = grep {/selenium_blank\d+/} $self->get_all_wind +ow_names; last if defined $window_name; $self->pause(100); } croak "Timed out waiting to select blank target window" if ! defin +ed $window_name; return $self->select_window($window_name); }

Setting Hidden Parameters

WWW::Mechanize has a number of things that are desirable. One of them is a convenient way to set hidden parameter values. I recently needed to do this and came up with the following using the get_eval() method which allows you to execute arbitrary JS.
sub set_hidden_parameter { my ($self, $form, $elem, $val) = @_; my $javascript = "window.document.$form.$elem.value = " . '"' . $v +al . '";'; my $rc = $self->get_eval('$javascript'); return $rc; }

Triggering JS Events

I ran across a problem where typing a value into a form via $sel->type($loc, $val); did not have the same affect as when doing it "manually". By looking at the source after the form was submitted, I realized that a hidden parameter was being set via some JS. Unfortunately, unraveling the JS initially proved too much for my feeble brain so I figured out how to set the hidden parameter by hand (see above). Eventually, I figured out that there was an onblur event. After that, the solution was quite easy:
$sel->type($loc, $val); $sel->fire_event($loc, 'onblur');

Adding Your Own Goodness

Subclassing WWW::Selenium isn't difficult. It uses LWP::UserAgent and HTTP::Request under the covers. I have been slowly adding much of the goodness in WWW::Mechanize. I will eventually release it but for now, it is an ugly hack.

This module is really deserving of a much larger FAQ and cookbook. I know it would have made things easier for me. I have been in contact with the module author who has been friendly and receptive. If you have any interest in this at all, I highly recommend you share that information.

Cheers - L~R

Replies are listed 'Best First'.
Re: Using WWW::Selenium To Test Or Automate An Ajax Website
by stonecolddevin (Parson) on Oct 28, 2008 at 19:25 UTC

    Limbic~Region++.

    Selenium is definitely a win. AFAIK Catalyst's test suite uses Selenium or at least makes use of Selenium, which makes things sooooo much easier.

    You should really use Selenium for web based testing.

    meh.
Re: Using WWW::Selenium To Test Or Automate An Ajax Website
by Limbic~Region (Chancellor) on Nov 20, 2008 at 23:40 UTC
    All,
    On 2008-11-20, I changed the heading on a couple of sections and added two new ones:
    • Setting Hidden Parameters
    • Triggering JS Events

    I have found getting help with Selenium problems hit or miss, so if you come across a "how do I" that you can't figure out, reply here and I will see what I can do.

    Cheers - L~R

      I'm having trouble verifying data in an Unordered List. I've tried using the Xpath as well as the Ids, but Selenium does not find them. Any advice would be very much appreciated. Thanks!
        Anonymous Monk,
        That question wasn't very specific so my response will be somewhat generic.

        First, did you use the IDE to record a session that selects the item from the unordered list? If so, does replaying yield the same results? If not - you should start there.

        Second, you say "... trouble verifying data in.." which leads me to believe you are not using the select() method but perhaps the get_select_options() method. It is hard to trouble shoot without knowing which method you are using and how are you using it (example)?

        Third, If there is truly some reason WWW::Selenium doesn't have a method you need, you have a number of options. You could use one of the myriad of HTML parsing modules on CPAN and parse it yourself. You could subclass it and fix which ever method is broken.

        Cheers - L~R

Re: Using WWW::Selenium To Test Or Automate An Ajax Website
by runrig (Abbot) on Mar 16, 2009 at 05:33 UTC

    I just want to mention that (possibly due to having later version(s) of some component(s)) I did get WWW::Selenium working using just "*firefox" as the browser setting. I did create a custom firefox profile in c:/selenium/firefox (which is still necessary due to one of the server startup options below), and at first used "*custom firefox..." like you had, but every time I started the Selenium server and launched Firefox, I had to create an exception for the SSL certificate (even with -trustAllSSLCertificates, and for some reason, I could not get "*firefox" working at first). Then I went back to using "*firefox", and everything worked! The only slightly annoying thing is that everytime Firefox starts, I get the add-on popup window with the message "3 new add-ons have been installed" ... not sure how to suppress that...have been looking for likely suspects.

    UPDATE: Actually, now I don't even need the -firefoxProfileTemplate or the -trustAllSSLCertificates options below, and I don't get the Add-on popup window. Also, I found the command to shutdown the Selenium RC server ('stop' only shuts Firefox down...do_command('shutDown') shuts the server down - and "do_command()" is undocumented AFAICT).

    Update: shutDown changed to shutDownSeleniumServer in newer versions of Selenium.

    I did add some options to the selenium server startup. Here is what the beginning of my program looks like:

    #!/usr/bin/perl use strict; use warnings; use WWW::Selenium; $ENV{PATH}="c:\\progra~1\\mozill~1;$ENV{PATH}"; # The first argument "1" is to make this run in the background # and not block on Windows. You would of course do this # differently on a *nix or other system. system(1, "java", "-jar", "c:/selenium/seleni~1.0-b/seleni~1.0-b/se23ae~1.0-b/seleni~1.jar", "-firefoxProfileTemplate", "c:/selenium/firefox", "-trustAllSSLCertificates", ); my $base = "https://xxxxx.xxxxx"; my $url = "$base/login/login.asp"; my $sel = WWW::Selenium->new( host => 'localhost', port => 4444, browser => "*firefox", # browser => '*custom firefox -no-remote -profile c:/selenium/firefo +x', browser_url => $base, ); $sel->start(); $sel->open($url); #And at the end... $sel->stop(); $sel->do_command('shutDownSeleniumServer');
      I have found same issue with SSL Please anybody guide me about it. Its urgent for me! my mail id is chetan_parmar7@hotmail.com Thanks, Chetan!
        What same issue? How are you launching Firefox? How are you launching the Selenium server? Show some code. Did you create a custom Firefox profile? No, I'm not emailing you, that's not how this site works, and it's not urgent to me.
Re: Using WWW::Selenium To Test Or Automate An Ajax Website
by spx2 (Deacon) on Dec 31, 2008 at 05:51 UTC

    Very very nice tutorial ! I was looking for something like this I was actually thinking of writing something similar with WWW::Selenium using Greasemonkey+Firefox+some_http_server_running_perl from "scratch" that would inject code in the page using AJAX to communicate between Perl and Javascript but now I'm more confident of WWW::Selenium s powers

    I have tried Selenium myself,installed it on my computer,it takes a pretty big hit on the resources as I remember...it was written in Java I think,it opened up it's own http server and had installed a firefox extension on my firefox.

    Basically the same thing can be done with Greasemonkey since it allows one to write code exactly after the page is rendered.

Re: Using WWW::Selenium To Test Or Automate An Ajax Website
by jdporter (Chancellor) on Aug 19, 2009 at 14:13 UTC
    Subclassing WWW::Selenium isn't difficult. ...
    I have been slowly adding much of the goodness in WWW::Mechanize. I will eventually release it but for now, it is an ugly hack.

    Mind if I inquire as to the status of that effort?

    Thanks for this great intro. Limbic~Region++ for sure.

      jdporter,
      It is still an ugly hack that is no longer under development. I was doing some freelance work using WWW::Selenium heavily. I have since dropped that project and since I love hate coding anything web related, it got dropped too. I would be happy to share with you what I do have if you want to make a go of it.

      Let me know as I don't want to spend the time extracting the proprietary implementation specific code unless you are interested.

      Cheers - L~R

Re: Using WWW::Selenium To Test Or Automate An Ajax Website
by jacques (Priest) on Nov 28, 2009 at 03:51 UTC
    Another solution is MozRepl, a FireFox plugin that allows you to interact with websites using JavaScript and Ajax.

    Luckily, there is a Perl interface for MozRepl on CPAN.

    I have also used Selenium and WWW::Selenium. They are really excellent products and indeed deserving of a much larger cookbook.

    One project to watch is Bromine. It's a web front-end for driving Selenium test cases. Unfortunately, it only officially supports Java and PHP, but there are reports of people using it with Perl test cases. The project is in its infancy and could use some attention.

Re: Using WWW::Selenium To Test Or Automate An Ajax Website
by icleave (Initiate) on Nov 24, 2009 at 16:44 UTC

    I think I may have the same port 4444 problem with some code I tried to write to learn how to use Selenium. Basically I tried to write a perl program on Ubuntu to automate the entry of cokezone codes on the www.cokezone.co.uk website. From my terminal I got the following results:

    icleave@icleave:~/Desktop/CokeZone$ perl cokezone.pl Error requesting http://localhost:4444/selenium-server/driver/?cmd=get +NewBrowserSession&1=*chrome&2=http%3A%2F%2Fwww.cokezone.co.uk%2F: 500 Can't connect to localhost:4444 (connect: Connection refused) # Looks like your test exited with 111 before it could output anything +.
    What advice do you have for sorting this out on Ubuntu? Many thanks, Icleave
      icleave,
      Are you running the Selenium core? You need to have Selenium running before you can use the Selenium Remote-Control (driven by WWW::Selenium. It should be along the lines of (from memory and likely off a bit):
      java --jar selenium-core.jar

      You can verify that you have something listening on port 4444 by doing a netstat -an | grep LISTEN | grep 4444. If you don't see anything then you haven't properly started Selenium.

      Cheers - L~R

        Ty Limbic~Region for the post. I wasn't aware I needed to run the Selenium core. I did try running $netstat -an | grep LISTEN | grep 4444 and it didn't do anything so I figured I really didn't have the Selenium core.

        I tried running the $java --jar selenium-core.jar suggestion from my terminal but I got the following error:

        Unrecognized option: --jar Could not create the Java virtual machine.
        I guessed you meant "-jar" rather than "--jar" so I retyped the command at the terminal but I got the following:

        The program 'java' can be found in the following packages: * gij-4.3 * java-gcj-compat-headless * openjdk-6-jre-headless * cacao * gij-4.2 * jamvm * kaffe Try: sudo apt-get install <selected package> bash: java: command not found

        So I tired "sudo apt-get selenium-core" hoping this would work but it didn't. So the next thing I did was try searching the Synaptic Package Manager with no luck. I then googled and found http://release.seleniumhq.org/selenium-core/1.0.1/selenium-core-1.0.1.zip. I downloaded it and extracted but I couldn't make heads or tails of what to do next after reading the 'install-readme.txt.'

        Any advice on how I actually install Selenium core on Ubuntu? All my other searches for information on this didn't result in anything I could use.

        Also assuming that I get Selenium core running, is there any code I need to know about to "start" the Selenium core? I can't remember where I read it, but I read that you need the following code in your perl program:

        $sel->start();

        If so, where do I put it? I put it in my program here:

        my $sel = Test::WWW::Selenium->new( host => "localhost", port => 4444, browser => "*chrome", browser_url => "http://www.example +.com/" ); $sel->start(); $sel->open("/home/index.jsp");

        Did I get this code position correct?

        Many thanks for the help!

        Nowadays, it is (apparently) recommended to use Selenium RC server instead — Selenium Core is no longer maintained. You can download it from http://seleniumhq.org/download/.

        To run it, do

        java -jar selenium-server.jar

        Make sure to use a recent enough version of Java, it fails here with java 1.4.2 but it works with 1.6.0.

Re: Using WWW::Selenium To Test Or Automate An Ajax Website
by sundialsvc4 (Abbot) on Apr 02, 2011 at 13:52 UTC
      Not to resurrect this thread too much (nor ones that I have raised before) ...

      Seekers of Perl Wisdom is that way, where new questions go :)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perltutorial [id://720018]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (3)
As of 2024-03-19 07:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found