Patrice's Weblog

This post about JavaScript testing is part of the Setting up a full stack for web testing series.

Introduction

JavaScript tests are sufficiently special to deserve a post outside of unit and functional testing. You need a special set of tools to write these tests. But then again for JavaScript the usual split between unit and functional tests is still applicable. So what I wrote about those two topics is also relevant here.

In this post I cover two separate tools for JavaScript testing: JsTestDriver, which is great for JavaScript unit tests, and Selenium which can be used to functional test a web application that uses JavaScript.

Selenium

Selenium is a software suite used to remotely control browsers. It consists of various sub-projects:

• Selenium Core: This is the original Selenium. It allows you to create a HTML page with some browser commands. Those commands can then be run in the browser when you open the page.
• Selenium IDE: A Firefox extension that can record your actions in the browser. This is a great way to learn the Selenium syntax as you can then export the recorded actions in several programming languages.
• Selenium Remote Control (Selenium RC): A Java server that allows controlling browsers remotely. Can be used from any programming language using a simple HTTP API.
• Selenium Grid: Basically the same as Selenium Remote Control but for running in parallel.

Personally I only use Selenium RC, so I won’t talk about the other parts. To get started, read the excellent Selenium documentation which explains everything a lot better than I could. If you want a quick start I recommend the Selenium RC chapter.

The basic test case in Selenium works like this:

1. Open the page to test (with by first logging in)
2. Execute some action like a click on a link or a mouse event
3. Wait for the action to execute (Ajax loads to finish, etc.)
4. Assert that the result is as expected

These four steps are repeated for every test case – so a lot of pages will be opened. Opening all those pages is the reason why Selenium tests tend to be very slow.

As a test framework you can use whatever you already have – in your favorite programming language. The difference is just that in your tests you will talk to Selenium to get the current state.

Take the following Python code as an example:

def test_login():
    sel = selenium("localhost", 4444, "*firefox", "http://localhost:5000/")
    sel.start()
    sel.open("/login")
    assert sel.is_text_present("Login")
    assert sel.is_element_present("username")
    assert sel.is_element_present("password")
    assert sel.get_value("username") == 'Username'

This script launches a Firefox browser and opens a login page of an application running on the localhost. It then gets several values from Selenium and asserts the correctness of these values using the standard test framework methods.

JsTestDriver

JsTestDriver is a relatively new tool which can be used to submit tests suites to browsers. Those browsers have to register with a Java-based server and you execute tests by submitting them to that same server.

So far that sounds very similar to Selenium. The difference is that JsTestDriver works with a blank page in which it directly inserts the test suite. It does that with a lot of optimizations to make sure the test runs are as fast as possible.

After that the unit test suite is run – directly inside the browser – and the client gets the test results including stack traces if available.

I recommend the Getting started documentation on the JsTestDriver wiki to see some code.

One of the main differences to Selenium is that you write the tests directly in JavaScript. There is a built-in test framework but you can also use other frameworks depending on your taste.

To show some code that you can contrast with the Selenium example above, consider this example:

GreeterTest = TestCase("GreeterTest");

GreeterTest.prototype.testGreet = function() {
  var greeter = new myapp.Greeter();
  assertEquals(“Hello World!”, greeter.greet(“World”));
};

Differences

Selenium is a very magic piece of software. Everybody falls in love with it when seeing their browser doing all the work. It’s something people just can’t resist. And so they end up using Selenium for all their web testing needs. I’ve been there myself.

The downside of Selenium is that it’s very brittle and slow. This is something that can’t really be avoided because all it does is control a browser. Opening pages and waiting for all scripts to load takes some time. And doing that hundreds or thousands of times, as is easily the case in a large test suite, leads to very slow test executions.

So instead of falling into that trap and only using Selenium, I recommend to clearly separate out unit tests which you can then execute in JsTestDriver. JsTestDriver does a lot less work and because of that is a lot more stable and faster. Then do integration tests with Selenium to test some basic workflows.

As an example take an autocompletion widget which is used on a search home page. Almost everything the widget does you can test by just calling its public interfaces and seeing if it does the right thing. This includes all the strange edge cases such as handling network timeouts or invalid data. So this part you do with a big JsTestDriver test suite. Then you only need one small functional test case to make sure the widget is correctly embedded in your home page. That’s your Selenium test.

As is evident I’m very happy that JsTestDriver has come along. Before that the only good solution for JavaScript testing was Selenium – and as I explained above it’s not a perfect fit for every testing need.

Conclusion

If you have followed my testing tutorial so far you now have all your testing tools set up. Some more chapters will follow but those now cover testing philosophy and tools around the principal testing framework.

Many developers see the value in getting started with testing their web
applications, but don’t know how to actually start using it. I’ll write about a variety of products and processes which cover the whole range of testing web applications. The posts in this series will be published in the coming weeks. You can find a rough table of contents below but the list may get modified or extended.

Approach

Initially you must spend some thoughts on how you want to approach testing. The main obstacle is that almost always you start with an existing project that already has a big code base. It seems impossible to ever get enough testing in place to be meaningful. But my experience shows that even very little coverage can already improve quality of your system a lot.

I recommend to enforce two very simple rules:

1. No bugfix without a test.
2. Develop new features using the test-driven methodology.

But whatever happens, don’t go and try to implement 100% code coverage for your legacy code. That will kill you – and drive motivation down very quickly.

Implementing a test for each bugfix is very little work. To fix the bug, you have to reproduce the problem anyway. During that process you will find at least one test case that can be implemented in an automated fashion relatively easy.

The first few tests will be the hardest. But don’t be discouraged by that, it will get easier. It’s hard mainly for two reasons. First and most important, testing is something you have to learn. You didn’t learn your programming language in just a few minutes either. Second a lot of software is not written with testability in mind.

I hope this series will help you getting better with both these aspects.

Table of Contents

This table of contents will get updated when I write the articles. Keep coming back to it.

1. Introduction
2. Testing framework
3. Unit testing
4. Functional testing
5. JavaScript testing
6. Continuous integration
7. Test-driven development
8. Testability
9. Measuring coverage

Maarten’s post at Tillate finally brought the motivation to document the PHP testing approach we use at local.ch.

First let me give you a short introduction to our architecture at local.ch. We have a clear separation of frontend (presentation, user-visible parts) and backend (search logic and database accesses). The frontend is written in PHP and XSLT. The PHP-part basically only orchestrates queries to our Java-based backend and passes the XML responses to XSLT. The bigger parts of the system are the XSLT stylesheet. All this means, that traditional unit tests don’t have a big value for the frontend as there isn’t much traditional logic. But we need to do functional/integration testing.

Only since a short time we actually have a nice PHP-based testing infrastructure. Before that, we almost exclusively used Selenium Core – see for example my presentation of last year. Now we use SimpleTest slightly extended and with a helper class for the Selenium testing (to be documented in a separate blog post).

This is the basic test.php file which we use to execute the tests:

require_once(“common.php”);


	
// “Configuration”

$GLOBALS[‘TLD’] = ‘local.ch’;

$GLOBALS[‘SELENIUM_SERVER’] = ‘localhost’;

	
if (file_exists(‘config_developer.php’)) {
    include_once(‘config_developer.php’);

}

if (getenv(‘SELENIUM_SERVER’)) {
    $GLOBALS[‘SELENIUM_SERVER’] = getenv(‘SELENIUM_SERVER’);

}

if (getenv(‘TLD’)) {
    $GLOBALS[‘TLD’] = getenv(‘TLD’);

}

	
/**
 * $case: Only run this test case
 * $test: Only run this test within the case
 */

function runAllTests($onlyCase = false, $onlyTest = false) {
    $test = &new TestSuite(‘All tests’);
    $dirs = array(“unit”, “selenium”, “selenium/*”);

    foreach ($dirs as $dir) {
        foreach (glob($dir . ‘/*.php’) as $file) {
            $test->addTestFile($file);
        }
    }

    if (!empty($onlyCase))
        $result = $test->run(new SelectiveReporter(new TextReporter(), $onlyCase, $onlyTest));
    else
        $result = $test->run(new XMLReporter());
    return ($result ? 0 : 1);
}

	
return runAllTests($argv[1], $argv2);

?>

The top part sets up some configuration values we use for Selenium. There are two global variables, the TLD which defines the host name to test against and SELENIUM_SERVER which is the Selenium server to connect to. There are two ways to configure. Either with the “config-developer.php” file which is excluded from version control and can be created by the developer. And then by setting environment variables when calling the test script.

After that the tests are run. Basically it includes tests from a set of directories. Then it either uses the SelectiveReporter or our own XMLReporter to execute tests. The SelectiveReporter will only execute a given test class or even only a given method (the first and second parameter from the command line respectively). The XMLReport gives a JUnit-style parseable output that we use for the continuous integration tool (Bamboo in our case).

The included common.php file contains this:

error_reporting(E_ALL);

ini_set(‘log_errors’, ‘0’);


	
if (! defined(‘SIMPLE_TEST’)) {
    define(‘SIMPLE_TEST’, BX_PROJECT_DIR . ‘/inc/vendor/simpletest/’);

}

require_once(SIMPLE_TEST . ‘reporter.php’);

require_once(SIMPLE_TEST . ‘unit_tester.php’);

	
class XMLReporter extends SimpleReporter {
    function XMLReporter() {
        $this->SimpleReporter();

        $this->doc = new DOMDocument();
        $this->doc->loadXML(’’);
        $this->root = $this->doc->documentElement;
    }

    function paintHeader($test_name) {
        $this->testsStart = microtime(true);

        $this->root->setAttribute(‘name’, $test_name);
        $this->root->setAttribute(‘timestamp’, date(‘c’));
        $this->root->setAttribute(‘hostname’, ‘localhost’);

        echo “\n”;
        echo “TLD’] . “ with selenium server “ . $GLOBALS[‘SELENIUM_SERVER’] . “\n”;
    }

    /**
     *    Paints the end of the test with a summary of
     *    the passes and failures.
     *    param string $test_name        Name class of test.
     *    access public
     */
    function paintFooter($test_name) {
        echo “—>\n”;

        $duration = microtime(true) – $this->testsStart;

        $this->root->setAttribute(‘tests’, $this->getPassCount() + $this->getFailCount() + $this->getExceptionCount());
        $this->root->setAttribute(‘failures’, $this->getFailCount());
        $this->root->setAttribute(‘errors’, $this->getExceptionCount());
        $this->root->setAttribute(‘time’, $duration);

        $this->doc->formatOutput = true;
        $xml = $this->doc->saveXML();
        // Cut out XML declaration
        echo preg_replace(‘/<\?[^>]*\?>/’, “”, $xml);
        echo “\n”;
    }

    function paintCaseStart($case) {
        echo “- case start $case\n”;
        $this->currentCaseName = $case;
    }

    function paintCaseEnd($case) {
        // No output here
    }

    function paintMethodStart($test) {
        echo “  – test start: $test\n”;

        $this->methodStart = microtime(true);
        $this->currCase = $this->doc->createElement(‘testcase’);
    }

    function paintMethodEnd($test) {
        $duration = microtime(true) – $this->methodStart;

        $this->currCase->setAttribute(‘name’, $test);
        $this->currCase->setAttribute(‘classname’, $this->currentCaseName);
        $this->currCase->setAttribute(‘time’, $duration);
        $this->root->appendChild($this->currCase);
    }

    function paintFail($message) {
        parent::paintFail($message);

        if (!$this->currCase) {
            error_log(“!! currCase was not set.”);
            return;
        }
        error_log(“Failure: “ . $message);

        $ch = $this->doc->createElement(‘failure’);
        $breadcrumb = $this->getTestList();
        $ch->setAttribute(‘message’, $breadcrumb[count($breadcrumb)-1]);
        $ch->setAttribute(‘type’, $breadcrumb[count($breadcrumb)-1]);

        $message = implode(’ -> ‘, $breadcrumb) . “\n\n\n” . $message;
        $content = $this->doc->createTextNode($message);
        $ch->appendChild($content);

        $this->currCase->appendChild($ch);
    }
}
?>

This file sets up SimpleTest by including the necessary file. Then follows the definition of the XMLReporter. It will print out some debugging so we know where it’s at. That’s necessary for us because our Selenium tests take about 15 to 20 minutes. At the end follows the XML-result which can be parsed by Bamboo. It should also work for other tools that expect JUnit XML output but I haven’t tested that.

For the next phase of local.ch E-Mail processes will play a central role. So I wanted to include those processes in our Selenium tests. It’s actually quite easy to do.

First create an account where test mails can go to. That account should be accessible by one of your scripts. I use a normal IMAP account for that. Then write a script which always outputs the newest mail on that account. I include some of the important headers plus the body (body parts for multi-part mails). I also made that page refresh itself every two seconds.

Then writing the tests is easy. Write a test first that executes the action that sends a mail. Make sure the mail is sent to your test account.

Next write a test that opens the getmail script (using the selenese command “open”). Follow that with a waitForTextPresent action to wait until the test mail has arrived – which never lasts more than a few seconds in my environment. Then you can use the normal test commands such as verifyText, verifyTextPresent or even click etc. if you output HTML mails correctly.

Works like a charm around here. If there is interest I can publish my script to get the mails. It’s written in PHP and is basically an IMAP client using the two PEAR packages Net_IMAP and Mail_mimeDecode.

Selenium is a open-source framework for functional testing of web applications. Today we found out the most likely reason for it’s naming.

Intervention to minimise the toxic effects of mercury and other free radical inducers is feasible for both the dental team and patients. Probably the most effective protective agent against the hazards of mercury poisoning is the element selenium.

Mercury apparently produces some test software that’s not well-liked with developers. Or so I gather as I have never worked with it myself.

Nice geek humour.

I spent Friday and Saturday at the T.Camp of the Swiss Web consultancy namics. That’s a yearly conference for the namics techies. As a member of the namics sister company local.ch I was also allowed to participate. And like every year for the past five years I held a presentation again. This time I talked about testing. Specifically Selenium.

It’s much the same as I’ll talk about on September 12 at the Webtuesday event.

You can download the presentation.