Paris: There is an App for that
I’ll quickly introduce the apps I used.
OffMaps
OffMaps is an iPhone map application using the OpenStreetMap data. That data is now surprisingly complete for many European cities and even includes many POIs such as restaurants, bars, public baths and many more.
OffMaps compiles that POI data into easy to use guides that also allow for offline search. Also you can download the full map of a guide or any area you choose to use it offline.
For the guides you need to pay extra. You can either buy single guides or – as I did – just buy the flat rate so you have great local information wherever you go.
The only information I missed was restaurant recommendations instead of only the locations. And routing does not work offline.
Memonic
Memonic is the online tool to take notes and save web site clippings I co-founded. Before the trip I saved the hotel reservation and some travel tips to Memonic.
The Memonic iPhone application downloads the whole collection for offline use. So in Paris I had access to all my Items without trouble.
Métro
All our traveling in Paris was either on foot or by public transport. Métro is a generic public transport app with data for many cities. Among others of course Paris. Once the data is downloaded you can get routes offline. I didn’t get to like the user interface during my few days of working with it but it always gave me correct answers very quickly.
Lonely Planet Paris City Guide
When the Eyjafjallajökull came upon us, Loney Planet decided to make some of their city guides available for free for some days. So I downloaded their Paris guide and used it in Paris.
In summary I didn’t like it at all. For starts, the information was too short, contained almost no links between different places and many copy-editing errors such as completely duplicated paragraphs made it into the product. But the real killer was lack of access. The app relies too much on search and doesn’t offer a good browsing access to find interesting places.
It does contain some restaurant recommendations and I did try one of those and liked it. But I ended up relying more on OffMaps even for that task.
Stanza
Stanza is an E-Book reader for the iPhone. It provided me with reading material for the train rides.
Built-ins
I also relied heavily on built-in applications. Calendar for storing itinerary information, Contacts for knowing where to send postcards to, Notes for some on-the-road information and the Clock for waking up in the morning.
Swiss Puppet user group
The first time we’ll have some lightning talks. I’ll do a presentation about my personal Puppet setup on my Mac workstations.
Get all the details on the official SPUG web site.
JavaScript testing
Introduction
JavaScript tests are sufficiently special to deserve a post outside of unit and functional testing. You need a special set of tools to write these tests. But then again for JavaScript the usual split between unit and functional tests is still applicable. So what I wrote about those two topics is also relevant here.
In this post I cover two separate tools for JavaScript testing: JsTestDriver, which is great for JavaScript unit tests, and Selenium which can be used to functional test a web application that uses JavaScript.
Selenium
Selenium is a software suite used to remotely control browsers. It consists of various sub-projects:
- Selenium Core: This is the original Selenium. It allows you to create a HTML page with some browser commands. Those commands can then be run in the browser when you open the page.
- Selenium IDE: A Firefox extension that can record your actions in the browser. This is a great way to learn the Selenium syntax as you can then export the recorded actions in several programming languages.
- Selenium Remote Control (Selenium RC): A Java server that allows controlling browsers remotely. Can be used from any programming language using a simple HTTP API.
- Selenium Grid: Basically the same as Selenium Remote Control but for running in parallel.
Personally I only use Selenium RC, so I won’t talk about the other parts. To get started, read the excellent Selenium documentation which explains everything a lot better than I could. If you want a quick start I recommend the Selenium RC chapter.
The basic test case in Selenium works like this:
- Open the page to test (with by first logging in)
- Execute some action like a click on a link or a mouse event
- Wait for the action to execute (Ajax loads to finish, etc.)
- Assert that the result is as expected
These four steps are repeated for every test case – so a lot of pages will be opened. Opening all those pages is the reason why Selenium tests tend to be very slow.
As a test framework you can use whatever you already have – in your favorite programming language. The difference is just that in your tests you will talk to Selenium to get the current state.
Take the following Python code as an example:
def test_login():
sel = selenium("localhost", 4444, "*firefox", "http://localhost:5000/")
sel.start()
sel.open("/login")
assert sel.is_text_present("Login")
assert sel.is_element_present("username")
assert sel.is_element_present("password")
assert sel.get_value("username") == 'Username'
This script launches a Firefox browser and opens a login page of an application running on the localhost. It then gets several values from Selenium and asserts the correctness of these values using the standard test framework methods.
JsTestDriver
JsTestDriver is a relatively new tool which can be used to submit tests suites to browsers. Those browsers have to register with a Java-based server and you execute tests by submitting them to that same server.
So far that sounds very similar to Selenium. The difference is that JsTestDriver works with a blank page in which it directly inserts the test suite. It does that with a lot of optimizations to make sure the test runs are as fast as possible.
After that the unit test suite is run – directly inside the browser – and the client gets the test results including stack traces if available.
I recommend the Getting started documentation on the JsTestDriver wiki to see some code.
One of the main differences to Selenium is that you write the tests directly in JavaScript. There is a built-in test framework but you can also use other frameworks depending on your taste.
To show some code that you can contrast with the Selenium example above, consider this example:
GreeterTest = TestCase("GreeterTest");
GreeterTest.prototype.testGreet = function() {
var greeter = new myapp.Greeter();
assertEquals("Hello World!", greeter.greet("World"));
};
Differences
Selenium is a very magic piece of software. Everybody falls in love with it when seeing their browser doing all the work. It’s something people just can’t resist. And so they end up using Selenium for all their web testing needs. I’ve been there myself.
The downside of Selenium is that it’s very brittle and slow. This is something that can’t really be avoided because all it does is control a browser. Opening pages and waiting for all scripts to load takes some time. And doing that hundreds or thousands of times, as is easily the case in a large test suite, leads to very slow test executions.
So instead of falling into that trap and only using Selenium, I recommend to clearly separate out unit tests which you can then execute in JsTestDriver. JsTestDriver does a lot less work and because of that is a lot more stable and faster. Then do integration tests with Selenium to test some basic workflows.
As an example take an autocompletion widget which is used on a search home page. Almost everything the widget does you can test by just calling its public interfaces and seeing if it does the right thing. This includes all the strange edge cases such as handling network timeouts or invalid data. So this part you do with a big JsTestDriver test suite. Then you only need one small functional test case to make sure the widget is correctly embedded in your home page. That’s your Selenium test.
As is evident I’m very happy that JsTestDriver has come along. Before that the only good solution for JavaScript testing was Selenium – and as I explained above it’s not a perfect fit for every testing need.
Conclusion
If you have followed my testing tutorial so far you now have all your testing tools set up. Some more chapters will follow but those now cover testing philosophy and tools around the principal testing framework.
Yahoo! Blueprint for mobile sites
The goal of this mobile site was to get a broad device coverage. As we’ll create an application for the iPhone, it was especially important to get a nice site for all the other devices. During his talk Adrian mentioned the Yahoo! Blueprint framework and I was sold quickly.
The basic architecture is relatively simple:
The really nice thing is that as a developer you don’t have to worry about any of the device detection, HTML generation, CSS fiddling, etc. That’s all handled in the “Yahoo server” box.
For our internal evaluation I gathered the following list of advantages and disadvantages:
Advantages:
- Very broad device coverage
- Fast development
- Image compression done by Yahoo
- Less code
Disadvantages:
- Mobile frontend hosted by Yahoo without any SLA (but transparent to the users, we can still serve it using the m.memonic.com domain)
- No HTML (we have to convert HTML to XForms XML – that’s a problem for detail pages on Memonic)
- No custom design (though to some extent that’s coming next year)
- No integration in Google Analytics (but we probably can use Yahoo’s own statistics tool for the mobile part)
- Currently no price plan
Based on that list we decided to go with Blueprint for the moment. The architecture means, that whenever we have to migrate away we can probably re-use our existing code and build something similar to Yahoo! Blueprint ourselves. That would actually be a really nice open source project.
For now: thank you Yahoo! for offering a wonderful service.
Pot: WSGI
This Python on the Toilet issue is also available as PDF.
WSGI is a Python standard for how web servers can interact with web frameworks. It’s one of my favorite standards: it’s simple yet very powerful.
To write a WSGI web application you only need to create one function.
def my_app(environ, start_response):
start_response('200 OK', [('Content-Type', 'text/plain')])
return repr(environ)
The function receives a environ argument – a dictionary with all the environment variables. The start_response function can be called with the status line and a list of headers as soon as you’re ready to send output.
New Python releases contain the library wsgiref which can be used to get started quickly with a simple web server (that should not be used in production).
pre..from wsgiref.simple_server import make_server
httpd = make_server(’', 8000, my_app)
httpd.serve_forever()
Save these two snippets in a file e.g. myapp.py and execute it with Python. This will serve the sample application on port 8000. Try it out.
You don’t usually want to write your own application directly on top of WSGI. But most frameworks now implement WSGI which has led to better interoperability. If you still want to use WSGI directly, there are a ton of good tools such as WebOb, “Werkzeug” or “Paste”. I used WebOb to easily build a REST service framework called WsgiService.
For more information I recommend the specification for WSGI which is a good read.
This post is part of the Python on the toilet series.
Slides for testing presentation
I’m now making the slides (in German) and the code examples available.
- Slides (PDF, 1.5 MB)
- Python code examples (ZIP, 12 KB)
Testing presentation on Tuesday (November 3)
The content of the presentation will be based on my current Setting up a full stack for web testing series.
Functional testing
Introduction
In functional testing, also referred to system testing or blackbox testing, you don’t just test the individual units but the whole application. So you’re testing if all of the small units you built and tested actually work together.
For functional testing a web site I talk to the application in HTTP. As mentioned already in Unit testing, an example for an address book would be to create an address using the provided web form, then check that the detail page renders and that it shows up in a list or search result.
I explicitly exclude JavaScript tests from this category, as I’ll cover those later in a separate post. The reason I don’t include them in functional tests is that Javascript tests tend to be more fickle and I try to have as stable functional tests as possible.
Tools
For functional testing you should get a tool which allows you to test the HTTP and HTML part of your applications.
As with the generic testing framework this is something that your web framework probably already provides. So check your framework documentation, some pointers:
- Django
- Okapi
- Pylons (though they call it “unit testing” it’s what I refer to as functional testing)
- Ruby on Rails
If you’re outside of a framework there are still great tools that you can use:
- Java: HttpUnit
- PHP: SimpleTest Web tester
- Python: WebTest (works with any WSGI application)
I really like the simplicity of the WebTest solution and think that more such tools should be provided. HttpUnit seems to have the same capabilities (using ServletUnit) but SimpleTest Web tester can only work against URLs.
Ground rules
These are some rules I made for myself over the years. They aren’t definitive and to be honest I break most of them from time to time. But if you adhere to those rules your functional tests will be a lot more manageable.
In addition to these the general rules of unit testing also apply of course.
Isolate the application under test
Generally it’s very useful for the tests to run in the same process as the web application under test. This way in-memory databases can be used which makes test suites a lot more stable. At the same time this means that only your tests can access the application and change the state.
If you can’t use an in-memory application, at least try to start up the server at the beginning of the test suite. The worst thing you can do from a stability perspective is to test against some shared deployment of your site.
For example I’ve been responsible for testing a classifieds listing site in the past. The tests created a classified listing using the web form, then checked if it was in the search result, etc. All of this was running against a central test server which also was used by the developers to try out the current version of the code. While this mostly worked, from time to time the tests would fail because some user or another test run was changing the listings. This caused test failures that were very hard to reproduce and were bugs in the tests – not in the code.
Don’t mock
In functional tests you should mock as little as possible. You should really treat the system under test as a blackbox and not know about all the method calls it will execute. In this context, mocks will make your tests very fragile.
To isolate the system you may use test databases (using sqlite’s :memory: for example) or dummy services instead of mocks.
Use Page Objects
You probably want to test the HTML output. For that you’ll need some selectors such as XPath expressions or CSS selectors to access the page contents. You’ll rely on parts of the DOM structure such as element names, CSS classes or IDs for that. Instead of hard-coding those in every test, create page objects. So then when the page layout changes, you’ll only have to adjust the page objects, not every test.
A very simple way to create a page object is to have one class per page type and define property on that. For example:
from lxml import etree
class Page(object):
"""Page object base class. Instantiated with some HTML."""
def __init__(self, body):
self._body = body
self._doc = etree.fromstring(body)
def __str__(self):
return self._body
def _get_text(self, xpath):
return self._doc.xpath('string(' + xpath + ')').strip()
class DetailPage(Page):
def get_title(self):
return self._get_text('/html/head/title')
def get_page_title(self):
return self._get_text('//h1')
This is just a simple example of how to do this. It expects a valid XML document as input as it uses the lxml parser. You could instead use BeautifulSoup or something similar.
An example for using these page objects:
def test_get_item(self):
"""app is an instance of webtest.TestApp"""
response = app.get('/item/id')
page = DetailPage(response.body)
assert page.get_title() == 'Test page - Memonic'
assert page.get_page_title() == 'Test page'
This assumes an existing app object where the test executes a request on. It then checks that the head title and page title are of the expected value.
Conclusion
Functional tests can mean a lot of satisfaction as they test the actual application. Some of my best success moments are when doing test-driven development with functional tests without ever opening a browser. It really allows me to get in a state where I just focus on what I want to accomplish and stay in a very productive mode.
If you have followed my tutorial so far, you now have a solid testing framework, good unit tests and can now progress with testing the meat of your application with functional tests.
In the next installment I’ll cover testing of the Javascript in your application.
Barcamp Rapperswil
We’ll be talking about mostly geek-topics during the event – though the topics depend entirely on you as a participant. Come and participate. I’ll present as well, though I still have to find a good topic.
You can find all the details and inscribe yourself on the official BarCampRapperswil page.
I hope to see you there.
Unit testing
Introduction
A modern application is split up in many functions, classes or modules. In unit testing you test the individual components and make sure they work as specified. This contrasts with functional or integration testing where you test multiple components at once.
As an example let’s assume you’re writing a web based address book. There will be functions to get a list of addresses from the database, update an address in the database, verify credentials of a user who wants to log in, etc. All of these will be tested with unit tests. A functional test on the other hand would for example test the web frontend by creating an address using the web form and then check if it can by found by the integrated search. So a functional test will rely on many of the base units.
To be able to write good unit tests, testability is an important factor. It basically means that you write your units completely independent of each-other. Testability will be covered in detail in a later post.
Now let’s get started with unit testing. Nowadays the basic unit of abstraction is a class. So I usually write one unit test class for each class in the code.
Example
Again all examples are in Python, but should be easily translated into your programming language of choice.
For starters let’s assume we have this model class which handles database access for addresses:
class AddressModel(object):
"""This is a very incomplete address book model."""
def __init__(self, db):
"""Constructor. db is a sqlite3 object."""
self.db = db
def get_address(self, id):
sql = "SELECT name FROM adr WHERE id=?"
c = self.db.cursor()
c.execute(sql, (id,))
row = c.fetchone()
if row:
return {'name': row[0]}
else:
return None
def setup_db(self):
"""Create the database table."""
self.db.execute("""CREATE TABLE adr (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name VARCHAR(300)
)""")
Forgive the ugly code, but it’s the shortest working example I could come up with.
The database is explicitly passed into the model. This might seem strange but this kind of dependency injection is the best way to ensure testability. Also the model is responsible for creating the database model. This will come in very handy for tests where the best way to test is using an in-memory database.
Now let’s write a simple test case. For this demo I just appended the following code to the same file as the address model (test.py) – though in a real project I would of course separate those.
import unittest
import sqlite3
class TestAddressModel(unittest.TestCase):
def setUp(self):
self.db = sqlite3.connect(':memory:')
self.model = AddressModel(self.db)
self.model.setup_db()
def create_user(self):
"""Create a simple test user."""
self.db.execute("INSERT INTO adr (id, name) VALUES (2, 'Tester')")
def test_get_no_data(self):
"""Database is empty, get_address returns None."""
self.assertEqual(self.model.get_address(1), None)
def test_get_other_id(self):
"""Database has one entry. Requesting a different id returns None."""
self.create_user()
self.assertEqual(self.model.get_address(1), None)
def test_get_name(self):
"""Database has one entry, requesting that entry returns a dict."""
self.create_user()
self.assertEqual(self.model.get_address(2), {'name': 'Tester'})
def test_get_sql_injection(self):
"""Ensure SQL Injection is not possible. See BUG-123 for details."""
self.create_user()
self.assertEqual(self.model.get_address('id --'), None)
if __name__ == '__main__':
unittest.main()
The resulting file can be executed using python: python test.py
Ground rules
There are a few very simple rules I try to follow when writing unit tests and I wrote these tests accordingly.
1. Don’t repeat yourself (DRY)
This is a general coding guidelines, but one that should also apply to tests. Make liberal use of the setUp method that most testing frameworks provide. Also extract common test scenarios (fixtures) into methods or even external data files. The create_user method above is an example.
2. Document the tests
Each of your tests should be documented. Where you have an entry in your bug tracking system, make sure you link to that. This way when the bug pops up again, the next developer has a lot more context.
The docstrings I use here in Python are particularly useful as they get printed when the test fails. For example:
======================================================================
FAIL: Database has one entry, requesting that entry returns a dict.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/pneff/Desktop/test.py", line 50, in test_get_name
self.assertEqual(self.model.get_address(2), {'name': 'Tester1'})
AssertionError: {'name': u'Tester'} != {'name': 'Tester1'}
3. Keep your tests small
Each test method should be as small as possible. If you make them too big it will become very hard to find out what exactly is failing. As a rule of thumb you should call only one method from the unit you’re testing.
4. Isolate tests
Do not rely on the order your tests are executed in. This happens more often than you’d think. For example let’s take this test class:
db = sqlite3.connect(':memory:')
model = AddressModel(db)
model.setup_db()
class TestAddressModelBadlyWritten(unittest.TestCase):
def test_get_empty(self):
"""Database is empty, get_address returns None."""
self.assertEqual(model.get_address(2), None)
def test_get_name(self):
"""Database has one entry, requesting that entry returns a dict."""
db.execute("INSERT INTO adr (id, name) VALUES (2, 'Tester')")
self.assertEqual(model.get_address(2), {'name': 'Tester'})
On my machine this works at the moment. But when I rename test_get_name to test_get_no_data it stops working. So this is a test where isolation failed.
This example might be clear and easy to spot and is also easy to fix (set up the model in the setUp method as was done in the original example). But this problem may sometimes be hidden behind a few layers of abstraction. A few examples of where you can encounter this:
- You may be using a central database (the same sqlite file, MySQL database, etc.) for all tests.
- You may be using some global variables or static class variables in your code.
- A lot of code saves some state on the file system.
- When relying on a web service that service can store central state.
You can often solve those problems with dependency injection and by using mocks objects.
If your tests are properly isolated you can also parallelize them. Try for example the nose multiprocess plugin.
Conclusion
Writing tests for a big web application is not easy. But if you follow a few simple rules and properly isolate the core units you have a good foundation to build on.
On the basis of good unit tests you can then go on and write functional tests.