JavaScript testing
This post about JavaScript testing is part of the Setting up a full stack for web testing series.
Introduction
JavaScript tests are sufficiently special to deserve a post outside of unit and functional testing. You need a special set of tools to write these tests. But then again for JavaScript the usual split between unit and functional tests is still applicable. So what I wrote about those two topics is also relevant here.
In this post I cover two separate tools for JavaScript testing: JsTestDriver, which is great for JavaScript unit tests, and Selenium which can be used to functional test a web application that uses JavaScript.
Selenium
Selenium is a software suite used to remotely control browsers. It consists of various sub-projects:
- Selenium Core: This is the original Selenium. It allows you to create a HTML page with some browser commands. Those commands can then be run in the browser when you open the page.
- Selenium IDE: A Firefox extension that can record your actions in the browser. This is a great way to learn the Selenium syntax as you can then export the recorded actions in several programming languages.
- Selenium Remote Control (Selenium RC): A Java server that allows controlling browsers remotely. Can be used from any programming language using a simple HTTP API.
- Selenium Grid: Basically the same as Selenium Remote Control but for running in parallel.
Personally I only use Selenium RC, so I won’t talk about the other parts. To get started, read the excellent Selenium documentation which explains everything a lot better than I could. If you want a quick start I recommend the Selenium RC chapter.
The basic test case in Selenium works like this:
- Open the page to test (with by first logging in)
- Execute some action like a click on a link or a mouse event
- Wait for the action to execute (Ajax loads to finish, etc.)
- Assert that the result is as expected
These four steps are repeated for every test case – so a lot of pages will be opened. Opening all those pages is the reason why Selenium tests tend to be very slow.
As a test framework you can use whatever you already have – in your favorite programming language. The difference is just that in your tests you will talk to Selenium to get the current state.
Take the following Python code as an example:
def test_login():
sel = selenium("localhost", 4444, "*firefox", "http://localhost:5000/")
sel.start()
sel.open("/login")
assert sel.is_text_present("Login")
assert sel.is_element_present("username")
assert sel.is_element_present("password")
assert sel.get_value("username") == 'Username'
This script launches a Firefox browser and opens a login page of an application running on the localhost. It then gets several values from Selenium and asserts the correctness of these values using the standard test framework methods.
JsTestDriver
JsTestDriver is a relatively new tool which can be used to submit tests suites to browsers. Those browsers have to register with a Java-based server and you execute tests by submitting them to that same server.
So far that sounds very similar to Selenium. The difference is that JsTestDriver works with a blank page in which it directly inserts the test suite. It does that with a lot of optimizations to make sure the test runs are as fast as possible.
After that the unit test suite is run – directly inside the browser – and the client gets the test results including stack traces if available.
I recommend the Getting started documentation on the JsTestDriver wiki to see some code.
One of the main differences to Selenium is that you write the tests directly in JavaScript. There is a built-in test framework but you can also use other frameworks depending on your taste.
To show some code that you can contrast with the Selenium example above, consider this example:
GreeterTest = TestCase("GreeterTest");
GreeterTest.prototype.testGreet = function() {
var greeter = new myapp.Greeter();
assertEquals(“Hello World!”, greeter.greet(“World”));
};
Differences
Selenium is a very magic piece of software. Everybody falls in love with it when seeing their browser doing all the work. It’s something people just can’t resist. And so they end up using Selenium for all their web testing needs. I’ve been there myself.
The downside of Selenium is that it’s very brittle and slow. This is something that can’t really be avoided because all it does is control a browser. Opening pages and waiting for all scripts to load takes some time. And doing that hundreds or thousands of times, as is easily the case in a large test suite, leads to very slow test executions.
So instead of falling into that trap and only using Selenium, I recommend to clearly separate out unit tests which you can then execute in JsTestDriver. JsTestDriver does a lot less work and because of that is a lot more stable and faster. Then do integration tests with Selenium to test some basic workflows.
As an example take an autocompletion widget which is used on a search home page. Almost everything the widget does you can test by just calling its public interfaces and seeing if it does the right thing. This includes all the strange edge cases such as handling network timeouts or invalid data. So this part you do with a big JsTestDriver test suite. Then you only need one small functional test case to make sure the widget is correctly embedded in your home page. That’s your Selenium test.
As is evident I’m very happy that JsTestDriver has come along. Before that the only good solution for JavaScript testing was Selenium – and as I explained above it’s not a perfect fit for every testing need.
Conclusion
If you have followed my testing tutorial so far you now have all your testing tools set up. Some more chapters will follow but those now cover testing philosophy and tools around the principal testing framework.
Slides for testing presentation
As announced, I held a talk today about testing.
I’m now making the slides (in German) and the code examples available.
- Slides (PDF, 1.5 MB)
- Python code examples (ZIP, 12 KB)
Testing presentation on Tuesday (November 3)
This Tuesday, November 3, I’ll give a presentation about testing at the Internet Briefing in Zürich. The talk will be in German. If you are interested, inscribe yourself on the Internet Briefing site.
The content of the presentation will be based on my current Setting up a full stack for web testing series.
Functional testing
This post about unit testing is part of the Setting up a full stack for web testing series.
Introduction
In functional testing, also referred to system testing or blackbox testing, you don’t just test the individual units but the whole application. So you’re testing if all of the small units you built and tested actually work together.
For functional testing a web site I talk to the application in HTTP. As mentioned already in Unit testing, an example for an address book would be to create an address using the provided web form, then check that the detail page renders and that it shows up in a list or search result.
I explicitly exclude JavaScript tests from this category, as I’ll cover those later in a separate post. The reason I don’t include them in functional tests is that Javascript tests tend to be more fickle and I try to have as stable functional tests as possible.
Tools
For functional testing you should get a tool which allows you to test the HTTP and HTML part of your applications.
As with the generic testing framework this is something that your web framework probably already provides. So check your framework documentation, some pointers:
- Django
- Okapi
- Pylons (though they call it “unit testing” it’s what I refer to as functional testing)
- Ruby on Rails
If you’re outside of a framework there are still great tools that you can use:
- Java: HttpUnit
- PHP: SimpleTest Web tester
- Python: WebTest (works with any WSGI application)
I really like the simplicity of the WebTest solution and think that more such tools should be provided. HttpUnit seems to have the same capabilities (using ServletUnit) but SimpleTest Web tester can only work against URLs.
Ground rules
These are some rules I made for myself over the years. They aren’t definitive and to be honest I break most of them from time to time. But if you adhere to those rules your functional tests will be a lot more manageable.
In addition to these the general rules of unit testing also apply of course.
Isolate the application under test
Generally it’s very useful for the tests to run in the same process as the web application under test. This way in-memory databases can be used which makes test suites a lot more stable. At the same time this means that only your tests can access the application and change the state.
If you can’t use an in-memory application, at least try to start up the server at the beginning of the test suite. The worst thing you can do from a stability perspective is to test against some shared deployment of your site.
For example I’ve been responsible for testing a classifieds listing site in the past. The tests created a classified listing using the web form, then checked if it was in the search result, etc. All of this was running against a central test server which also was used by the developers to try out the current version of the code. While this mostly worked, from time to time the tests would fail because some user or another test run was changing the listings. This caused test failures that were very hard to reproduce and were bugs in the tests – not in the code.
Don’t mock
In functional tests you should mock as little as possible. You should really treat the system under test as a blackbox and not know about all the method calls it will execute. In this context, mocks will make your tests very fragile.
To isolate the system you may use test databases (using sqlite’s :memory: for example) or dummy services instead of mocks.
Use Page Objects
You probably want to test the HTML output. For that you’ll need some selectors such as XPath expressions or CSS selectors to access the page contents. You’ll rely on parts of the DOM structure such as element names, CSS classes or IDs for that. Instead of hard-coding those in every test, create page objects. So then when the page layout changes, you’ll only have to adjust the page objects, not every test.
A very simple way to create a page object is to have one class per page type and define property on that. For example:
from lxml import etree
class Page(object):
“”“Page object base class. Instantiated with some <span class="caps">HTML</span>.”“”
def __init__(self, body):
self._body = body
self._doc = etree.fromstring(body)
def __str__(self):
return self._body
def _get_text(self, xpath):
return self._doc.xpath(‘string(’ + xpath + ‘)’).strip()
class DetailPage(Page):
def get_title(self):
return self._get_text(‘/html/head/title’)
def get_page_title(self):
return self._get_text(‘//h1’)
This is just a simple example of how to do this. It expects a valid XML document as input as it uses the lxml parser. You could instead use BeautifulSoup or something similar.
An example for using these page objects:
def test_get_item(self):
"""app is an instance of webtest.TestApp"""
response = app.get('/item/id')
page = DetailPage(response.body)
assert page.get_title() == 'Test page - Memonic'
assert page.get_page_title() == 'Test page'
This assumes an existing app object where the test executes a request on. It then checks that the head title and page title are of the expected value.
Conclusion
Functional tests can mean a lot of satisfaction as they test the actual application. Some of my best success moments are when doing test-driven development with functional tests without ever opening a browser. It really allows me to get in a state where I just focus on what I want to accomplish and stay in a very productive mode.
If you have followed my tutorial so far, you now have a solid testing framework, good unit tests and can now progress with testing the meat of your application with functional tests.
In the next installment I’ll cover testing of the Javascript in your application.
Unit testing
This post about unit testing is part of the Setting up a full stack for web testing series.
Introduction
A modern application is split up in many functions, classes or modules. In unit testing you test the individual components and make sure they work as specified. This contrasts with functional or integration testing where you test multiple components at once.
As an example let’s assume you’re writing a web based address book. There will be functions to get a list of addresses from the database, update an address in the database, verify credentials of a user who wants to log in, etc. All of these will be tested with unit tests. A functional test on the other hand would for example test the web frontend by creating an address using the web form and then check if it can by found by the integrated search. So a functional test will rely on many of the base units.
To be able to write good unit tests, testability is an important factor. It basically means that you write your units completely independent of each-other. Testability will be covered in detail in a later post.
Now let’s get started with unit testing. Nowadays the basic unit of abstraction is a class. So I usually write one unit test class for each class in the code.
Example
Again all examples are in Python, but should be easily translated into your programming language of choice.
For starters let’s assume we have this model class which handles database access for addresses:
class AddressModel(object):
"""This is a very incomplete address book model."""
def __init__(self, db):
"""Constructor. db is a sqlite3 object."""
self.db = db
def get_address(self, id):
sql = “<span class="caps">SELECT</span> name <span class="caps">FROM</span> adr <span class="caps">WHERE</span> id=?”
c = self.db.cursor()
c.execute(sql, (id,))
row = c.fetchone()
if row:
return {‘name’: row<sup class="footnote" id="fnreve0c1350929f445f48b787cf09cedbf2f-1"><a href="#fne0c1350929f445f48b787cf09cedbf2f-1">0</a></sup>}
else:
return None
def setup_db(self):
“”“Create the database table.”“”
self.db.execute(“”“<span class="caps">CREATE</span> <span class="caps">TABLE</span> adr (
id <span class="caps">INTEGER</span> <span class="caps">PRIMARY</span> <span class="caps">KEY</span> <span class="caps">AUTOINCREMENT</span>,
name <acronym title="300"><span class="caps">VARCHAR</span></acronym>
)”“”)
Forgive the ugly code, but it’s the shortest working example I could come up with.
The database is explicitly passed into the model. This might seem strange but this kind of dependency injection is the best way to ensure testability. Also the model is responsible for creating the database model. This will come in very handy for tests where the best way to test is using an in-memory database.
Now let’s write a simple test case. For this demo I just appended the following code to the same file as the address model (test.py) – though in a real project I would of course separate those.
import unittest
import sqlite3
class TestAddressModel(unittest.TestCase):
def setUp(self):
self.db = sqlite3.connect(‘:memory:’)
self.model = AddressModel(self.db)
self.model.setup_db()
def create_user(self):
“”“Create a simple test user.”“”
self.db.execute(“<span class="caps">INSERT</span> <span class="caps">INTO</span> adr (id, name) <span class="caps">VALUES</span> (2, ‘Tester’)”)
def test_get_no_data(self):
“”“Database is empty, get_address returns None.”“”
self.assertEqual(self.model.get_address(1), None)
def test_get_other_id(self):
“”“Database has one entry. Requesting a different id returns None.”“”
self.create_user()
self.assertEqual(self.model.get_address(1), None)
def test_get_name(self):
“”“Database has one entry, requesting that entry returns a dict.”“”
self.create_user()
self.assertEqual(self.model.get_address(2), {‘name’: ‘Tester’})
def test_get_sql_injection(self):
“”“Ensure <span class="caps">SQL</span> Injection is not possible. See <span class="caps">BUG</span>-123 for details.”“”
self.create_user()
self.assertEqual(self.model.get_address(‘id —’), None)
if <i>name</i> == ‘<i>main</i>’:
unittest.main()
The resulting file can be executed using python: python test.py
Ground rules
There are a few very simple rules I try to follow when writing unit tests and I wrote these tests accordingly.
1. Don’t repeat yourself (DRY)
This is a general coding guidelines, but one that should also apply to tests. Make liberal use of the setUp method that most testing frameworks provide. Also extract common test scenarios (fixtures) into methods or even external data files. The create_user method above is an example.
2. Document the tests
Each of your tests should be documented. Where you have an entry in your bug tracking system, make sure you link to that. This way when the bug pops up again, the next developer has a lot more context.
The docstrings I use here in Python are particularly useful as they get printed when the test fails. For example:
======================================================================
FAIL: Database has one entry, requesting that entry returns a dict.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/pneff/Desktop/test.py", line 50, in test_get_name
self.assertEqual(self.model.get_address(2), {'name': 'Tester1'})
AssertionError: {'name': u'Tester'} != {'name': 'Tester1'}
3. Keep your tests small
Each test method should be as small as possible. If you make them too big it will become very hard to find out what exactly is failing. As a rule of thumb you should call only one method from the unit you’re testing.
4. Isolate tests
Do not rely on the order your tests are executed in. This happens more often than you’d think. For example let’s take this test class:
db = sqlite3.connect(':memory:')
model = AddressModel(db)
model.setup_db()
class TestAddressModelBadlyWritten(unittest.TestCase):
def test_get_empty(self):
“”“Database is empty, get_address returns None.”“”
self.assertEqual(model.get_address(2), None)
def test_get_name(self):
“”“Database has one entry, requesting that entry returns a dict.”“”
db.execute(“<span class="caps">INSERT</span> <span class="caps">INTO</span> adr (id, name) <span class="caps">VALUES</span> (2, ‘Tester’)”)
self.assertEqual(model.get_address(2), {‘name’: ‘Tester’})
On my machine this works at the moment. But when I rename test_get_name to test_get_no_data it stops working. So this is a test where isolation failed.
This example might be clear and easy to spot and is also easy to fix (set up the model in the setUp method as was done in the original example). But this problem may sometimes be hidden behind a few layers of abstraction. A few examples of where you can encounter this:
- You may be using a central database (the same sqlite file, MySQL database, etc.) for all tests.
- You may be using some global variables or static class variables in your code.
- A lot of code saves some state on the file system.
- When relying on a web service that service can store central state.
You can often solve those problems with dependency injection and by using mocks objects.
If your tests are properly isolated you can also parallelize them. Try for example the nose multiprocess plugin.
Conclusion
Writing tests for a big web application is not easy. But if you follow a few simple rules and properly isolate the core units you have a good foundation to build on.
On the basis of good unit tests you can then go on and write functional tests.
Testing framework
This post is part of the Setting up a full stack for web testing series.
The initial setup of the testing framework is very crucial. You need to make it as easy and fast as possible for all developers to run the tests. You also need to make sure those tests are executed automatically even when developers become too lazy. That second part I’ll cover in the continuous integration post at a later stage.
In my opinion, setting up a good framework is the most important work you do when it comes to testing. Any work you do here will be multiplied later when you and your team spend less time on writing the tests.
I recommend you start by just adding a tests folder to your project and include one empty feel-good test. See below for a Python example of such a test.
Most modern web frameworks already include a complete framework ready to run. Ruby on Rails started that trend and almost every modern MVC framework comes with a similar testing framework. So chances are that all the hard work has already been done for you. Refer to your framework’s documentation for details.
If you aren’t using a web framework, here are some pointers for getting started outside of that context:
For all of these tools I tried to find a tutorial that shows how to set up a full test, though they aren’t all very complete. If the tutorial doesn’t cover that part yet, spend some additional time to make the tests runnable on the command line with just one command. To achieve that, the Ruby example above uses Rake, nose is already a one-stop testing tool and Java developers commonly use Ant. In PHP projects I have used Rake as well, to run all the tests in a project easily.
If your work mainly in an IDE, you may want to get good integration to execute tests easily as well. I can’t give you good pointers for that, as the configuration is very different between IDEs. But you’ll easily find the required information with any search engine.
The tests I usually include to make sure the framework is up and running are these two:
def test_true():
assert True
def test_false():
assert False
This is Python code to be run with nose. But translating that to your framework won’t be a problem.
If you have good tutorials for other languages and frameworks, please leave them in the comments so I can include them in the article.
Setting up a full stack for web testing
Many developers see the value in getting started with testing their web
applications, but don’t know how to actually start using it. I’ll write about a variety of products and processes which cover the whole range of testing web applications. The posts in this series will be published in the coming weeks. You can find a rough table of contents below but the list may get modified or extended.
Approach
Initially you must spend some thoughts on how you want to approach testing. The main obstacle is that almost always you start with an existing project that already has a big code base. It seems impossible to ever get enough testing in place to be meaningful. But my experience shows that even very little coverage can already improve quality of your system a lot.
I recommend to enforce two very simple rules:
- No bugfix without a test.
- Develop new features using the test-driven methodology.
But whatever happens, don’t go and try to implement 100% code coverage for your legacy code. That will kill you – and drive motivation down very quickly.
Implementing a test for each bugfix is very little work. To fix the bug, you have to reproduce the problem anyway. During that process you will find at least one test case that can be implemented in an automated fashion relatively easy.
The first few tests will be the hardest. But don’t be discouraged by that, it will get easier. It’s hard mainly for two reasons. First and most important, testing is something you have to learn. You didn’t learn your programming language in just a few minutes either. Second a lot of software is not written with testability in mind.
I hope this series will help you getting better with both these aspects.
Table of Contents
This table of contents will get updated when I write the articles. Keep coming back to it.
- Introduction
- Testing framework
- Unit testing
- Functional testing
- JavaScript testing
- Continuous integration
- Test-driven development
- Testability
- Measuring coverage
Pot: nose
nose is my Python testing tool of choice. You can use it to very easily create test suites.
While it does support the unittest module which is included with Python, it also allows for a much simple style using just methods and asserts.
Put the following code into a file called test_basic.py:
def test_addition():
print 2 + 2
assert 2 + 2 == 4
def test_addition_wrong():
print 2 + 3
assert 2 + 3 == 6
Then just run nosetests in the directory where you stored this file. You’ll
get an output indicating, that there is a test failure.
nose is based completely on naming conventions. Make sure your test files and methods all start with test for them to get executed. For assertions, just use assert.
When a test goes wrong you usually need some additional output to know what happened. nose helps you with this by handling print statements nicely. In the example above, as test_addition_wrong fails, you’ll see the output of the print statement print 2 + 3. But the print output of test_addition is suppressed because that test does not fail.
One of my favourite features of nose is Test generators. Often you have a large list of inputs and corresponding expected outputs for a function. Test generators allow you to easily test all of them without having to repeat the test code many times.
An example from a Nektoon test suite:
def test_browsers():
tests = [('Mozilla/4.0 (compatible; MSIE 8.0)', 'msie', '8'),
('Mozilla/4.0 (compatible; MSIE 7.0b)', 'msie', '7')]
for ua, model, version in tests:
yield check_browser, ua, model, version
def check_browser(ua, model, version):
res = parse_ua(ua)
print res
assert res['model'] == model
assert res['version'] == version
The tests list is of course much bigger in our test suite and that’s the beauty of it. Adding a new test is very easy, but nose still provides very meaningful error output in the case of a test failure.
Read the official nose documentation if I was able to wet your appetite.
This post is part of the Python on the toilet series.
Continuous testing with Python
Back when I did some Ruby on Rails development I was a big fan of autotest. With it I could stay in my editor while the project test suite got executed with every change.
Setup
Now that I’m working in Python I was looking for something similar and I was successful. You’ll need the following:
Then you can execute tdaemon like this in your project directory:
Usage
tdaemon.py --custom-args='--with-growl'
This will continually execute the test suite and notify you with growl about the status as you can see in this screencast:
Demo
Details
To get this working as shown in the screencast I actually had to make some changes.
First nosegrowl didn’t install well using easy_install as the images were missing. So I went ahead and did it manually:
$ hg clone http://hg.assembla.com/nosegrowl $ cd nosegrowl/nose-growl/ $ sed -i.bak 's/growl.start/# growl.start/' nosegrowl/growler.py $ python setup.py install
The ‘sed’ command is optional. But I don’t want to be notified when the test suite starts, only when it ends. So I uncomment the growl.start line.
Additionally to make tdaemon less noisy when working with vim I added the swap files to the exclude list. Open the tdaemon.py file and edit the IGNORE_EXTENSIONS line to look like this:
IGNORE_EXTENSIONS = ('pyc', 'pyo', 'swp')A/B Testing presentation
I spent the day at StartupCamp in Basel yesterday. I thoroughly enjoyed the conference and would love to see it happen again.
This being a BarCamp I did a presentation as well. Mine was about A/B testing or multivariate testing. It’s not exactly the same thing but I tend to lump them together.
I collected all the background material on delicious with the abtest tag.
You can download the presentation (PDF, 2MB). There were some more nice pictures in my live presentation but I stripped them for online consumption for copyright reasons.
PHP Testing with SimpleTest
Maarten’s post at Tillate finally brought the motivation to document the PHP testing approach we use at local.ch.
First let me give you a short introduction to our architecture at local.ch. We have a clear separation of frontend (presentation, user-visible parts) and backend (search logic and database accesses). The frontend is written in PHP and XSLT. The PHP-part basically only orchestrates queries to our Java-based backend and passes the XML responses to XSLT. The bigger parts of the system are the XSLT stylesheet. All this means, that traditional unit tests don’t have a big value for the frontend as there isn’t much traditional logic. But we need to do functional/integration testing.
Only since a short time we actually have a nice PHP-based testing infrastructure. Before that, we almost exclusively used Selenium Core – see for example my presentation of last year. Now we use SimpleTest slightly extended and with a helper class for the Selenium testing (to be documented in a separate blog post).
This is the basic test.php file which we use to execute the tests:
require_once(“common.php”);
// “Configuration”
$GLOBALS[‘TLD’] = ‘local.ch’;
$GLOBALS[‘SELENIUM_SERVER’] = ‘localhost’;
if (file_exists(‘config_developer.php’)) {
include_once(‘config_developer.php’);
}
if (getenv(‘SELENIUM_SERVER’)) {
$GLOBALS[‘SELENIUM_SERVER’] = getenv(‘SELENIUM_SERVER’);
}
if (getenv(‘TLD’)) {
$GLOBALS[‘TLD’] = getenv(‘TLD’);
}
/**
* $case: Only run this test case
* $test: Only run this test within the case
*/
function runAllTests($onlyCase = false, $onlyTest = false) {
$test = &new TestSuite(‘All tests’);
$dirs = array(“unit”, “selenium”, “selenium/*”);
foreach ($dirs as $dir) {
foreach (glob($dir . ‘/*.php’) as $file) {
$test->addTestFile($file);
}
}
if (!empty($onlyCase))
$result = $test->run(new SelectiveReporter(new TextReporter(), $onlyCase, $onlyTest));
else
$result = $test->run(new XMLReporter());
return ($result ? 0 : 1);
}
return runAllTests($argv[1], $argv2);
?>
The top part sets up some configuration values we use for Selenium. There are two global variables, the TLD which defines the host name to test against and SELENIUM_SERVER which is the Selenium server to connect to. There are two ways to configure. Either with the “config-developer.php” file which is excluded from version control and can be created by the developer. And then by setting environment variables when calling the test script.
After that the tests are run. Basically it includes tests from a set of directories. Then it either uses the SelectiveReporter or our own XMLReporter to execute tests. The SelectiveReporter will only execute a given test class or even only a given method (the first and second parameter from the command line respectively). The XMLReport gives a JUnit-style parseable output that we use for the continuous integration tool (Bamboo in our case).
The included common.php file contains this:
error_reporting(E_ALL);
ini_set(‘log_errors’, ‘0’);
if (! defined(‘SIMPLE_TEST’)) {
define(‘SIMPLE_TEST’, BX_PROJECT_DIR . ‘/inc/vendor/simpletest/’);
}
require_once(SIMPLE_TEST . ‘reporter.php’);
require_once(SIMPLE_TEST . ‘unit_tester.php’);
class XMLReporter extends SimpleReporter {
function XMLReporter() {
$this->SimpleReporter();
$this->doc = new DOMDocument();
$this->doc->loadXML(’param string $test_name Name class of test.
* access public
*/
function paintFooter($test_name) {
echo “—>\n”;
$duration = microtime(true) – $this->testsStart;
$this->root->setAttribute(‘tests’, $this->getPassCount() + $this->getFailCount() + $this->getExceptionCount());
$this->root->setAttribute(‘failures’, $this->getFailCount());
$this->root->setAttribute(‘errors’, $this->getExceptionCount());
$this->root->setAttribute(‘time’, $duration);
$this->doc->formatOutput = true;
$xml = $this->doc->saveXML();
// Cut out XML declaration
echo preg_replace(‘/<\?[^>]*\?>/’, “”, $xml);
echo “\n”;
}
function paintCaseStart($case) {
echo “- case start $case\n”;
$this->currentCaseName = $case;
}
function paintCaseEnd($case) {
// No output here
}
function paintMethodStart($test) {
echo “ – test start: $test\n”;
$this->methodStart = microtime(true);
$this->currCase = $this->doc->createElement(‘testcase’);
}
function paintMethodEnd($test) {
$duration = microtime(true) – $this->methodStart;
$this->currCase->setAttribute(‘name’, $test);
$this->currCase->setAttribute(‘classname’, $this->currentCaseName);
$this->currCase->setAttribute(‘time’, $duration);
$this->root->appendChild($this->currCase);
}
function paintFail($message) {
parent::paintFail($message);
if (!$this->currCase) {
error_log(“!! currCase was not set.”);
return;
}
error_log(“Failure: “ . $message);
$ch = $this->doc->createElement(‘failure’);
$breadcrumb = $this->getTestList();
$ch->setAttribute(‘message’, $breadcrumb[count($breadcrumb)-1]);
$ch->setAttribute(‘type’, $breadcrumb[count($breadcrumb)-1]);
$message = implode(’ -> ‘, $breadcrumb) . “\n\n\n” . $message;
$content = $this->doc->createTextNode($message);
$ch->appendChild($content);
$this->currCase->appendChild($ch);
}
}
?>This file sets up SimpleTest by including the necessary file. Then follows the definition of the XMLReporter. It will print out some debugging so we know where it’s at. That’s necessary for us because our Selenium tests take about 15 to 20 minutes. At the end follows the XML-result which can be parsed by Bamboo. It should also work for other tools that expect JUnit XML output but I haven’t tested that.
Mail testing with Selenium
For the next phase of local.ch E-Mail processes will play a central role. So I wanted to include those processes in our Selenium tests. It’s actually quite easy to do.
First create an account where test mails can go to. That account should be accessible by one of your scripts. I use a normal IMAP account for that. Then write a script which always outputs the newest mail on that account. I include some of the important headers plus the body (body parts for multi-part mails). I also made that page refresh itself every two seconds.
Then writing the tests is easy. Write a test first that executes the action that sends a mail. Make sure the mail is sent to your test account.
Next write a test that opens the getmail script (using the selenese command “open”). Follow that with a waitForTextPresent action to wait until the test mail has arrived – which never lasts more than a few seconds in my environment. Then you can use the normal test commands such as verifyText, verifyTextPresent or even click etc. if you output HTML mails correctly.
Works like a charm around here. If there is interest I can publish my script to get the mails. It’s written in PHP and is basically an IMAP client using the two PEAR packages Net_IMAP and Mail_mimeDecode.
Name of Selenium
Selenium is a open-source framework for functional testing of web applications. Today we found out the most likely reason for it’s naming.
Intervention to minimise the toxic effects of mercury and other free radical inducers is feasible for both the dental team and patients. Probably the most effective protective agent against the hazards of mercury poisoning is the element selenium.
Mercury apparently produces some test software that’s not well-liked with developers. Or so I gather as I have never worked with it myself.
Nice geek humour.
Testing with Selenium
I spent Friday and Saturday at the T.Camp of the Swiss Web consultancy namics. That’s a yearly conference for the namics techies. As a member of the namics sister company local.ch I was also allowed to participate. And like every year for the past five years I held a presentation again. This time I talked about testing. Specifically Selenium.
It’s much the same as I’ll talk about on September 12 at the Webtuesday event.
You can download the presentation.