Yahoo! Blueprint for mobile sites
The goal of this mobile site was to get a broad device coverage. As we’ll create an application for the iPhone, it was especially important to get a nice site for all the other devices. During his talk Adrian mentioned the Yahoo! Blueprint framework and I was sold quickly.
The basic architecture is relatively simple:
The really nice thing is that as a developer you don’t have to worry about any of the device detection, HTML generation, CSS fiddling, etc. That’s all handled in the “Yahoo server” box.
For our internal evaluation I gathered the following list of advantages and disadvantages:
Advantages:
- Very broad device coverage
- Fast development
- Image compression done by Yahoo
- Less code
Disadvantages:
- Mobile frontend hosted by Yahoo without any SLA (but transparent to the users, we can still serve it using the m.memonic.com domain)
- No HTML (we have to convert HTML to XForms XML – that’s a problem for detail pages on Memonic)
- No custom design (though to some extent that’s coming next year)
- No integration in Google Analytics (but we probably can use Yahoo’s own statistics tool for the mobile part)
- Currently no price plan
Based on that list we decided to go with Blueprint for the moment. The architecture means, that whenever we have to migrate away we can probably re-use our existing code and build something similar to Yahoo! Blueprint ourselves. That would actually be a really nice open source project.
For now: thank you Yahoo! for offering a wonderful service.
Pot: WSGI
This Python on the Toilet issue is also available as PDF.
WSGI is a Python standard for how web servers can interact with web frameworks. It’s one of my favorite standards: it’s simple yet very powerful.
To write a WSGI web application you only need to create one function.
def my_app(environ, start_response):
start_response('200 OK', [('Content-Type', 'text/plain')])
return repr(environ)
The function receives a environ argument – a dictionary with all the environment variables. The start_response function can be called with the status line and a list of headers as soon as you’re ready to send output.
New Python releases contain the library wsgiref which can be used to get started quickly with a simple web server (that should not be used in production).
pre..from wsgiref.simple_server import make_server
httpd = make_server(’', 8000, my_app)
httpd.serve_forever()
Save these two snippets in a file e.g. myapp.py and execute it with Python. This will serve the sample application on port 8000. Try it out.
You don’t usually want to write your own application directly on top of WSGI. But most frameworks now implement WSGI which has led to better interoperability. If you still want to use WSGI directly, there are a ton of good tools such as WebOb, “Werkzeug” or “Paste”. I used WebOb to easily build a REST service framework called WsgiService.
For more information I recommend the specification for WSGI which is a good read.
This post is part of the Python on the toilet series.
Slides for testing presentation
I’m now making the slides (in German) and the code examples available.
- Slides (PDF, 1.5 MB)
- Python code examples (ZIP, 12 KB)
Testing presentation on Tuesday (November 3)
The content of the presentation will be based on my current Setting up a full stack for web testing series.
Functional testing
Introduction
In functional testing, also referred to system testing or blackbox testing, you don’t just test the individual units but the whole application. So you’re testing if all of the small units you built and tested actually work together.
For functional testing a web site I talk to the application in HTTP. As mentioned already in Unit testing, an example for an address book would be to create an address using the provided web form, then check that the detail page renders and that it shows up in a list or search result.
I explicitly exclude Javascript tests from this category, as I’ll cover those later in a separate post. The reason I don’t include them in functional tests is that Javascript tests tend to be more fickle and I try to have as stable functional tests as possible.
Tools
For functional testing you should get a tool which allows you to test the HTTP and HTML part of your applications.
As with the generic testing framework this is something that your web framework probably already provides. So check your framework documentation, some pointers:
- Django
- Okapi
- Pylons (though they call it “unit testing” it’s what I refer to as functional testing)
- Ruby on Rails
If you’re outside of a framework there are still great tools that you can use:
- Java: HttpUnit
- PHP: SimpleTest Web tester
- Python: WebTest (works with any WSGI application)
I really like the simplicity of the WebTest solution and think that more such tools should be provided. HttpUnit seems to have the same capabilities (using ServletUnit) but SimpleTest Web tester can only work against URLs.
Ground rules
These are some rules I made for myself over the years. They aren’t definitive and to be honest I break most of them from time to time. But if you adhere to those rules your functional tests will be a lot more manageable.
In addition to these the general rules of unit testing also apply of course.
Isolate the application under test
Generally it’s very useful for the tests to run in the same process as the web application under test. This way in-memory databases can be used which makes test suites a lot more stable. At the same time this means that only your tests can access the application and change the state.
If you can’t use an in-memory application, at least try to start up the server at the beginning of the test suite. The worst thing you can do from a stability perspective is to test against some shared deployment of your site.
For example I’ve been responsible for testing a classifieds listing site in the past. The tests created a classified listing using the web form, then checked if it was in the search result, etc. All of this was running against a central test server which also was used by the developers to try out the current version of the code. While this mostly worked, from time to time the tests would fail because some user or another test run was changing the listings. This caused test failures that were very hard to reproduce and were bugs in the tests – not in the code.
Don’t mock
In functional tests you should mock as little as possible. You should really treat the system under test as a blackbox and not know about all the method calls it will execute. In this context, mocks will make your tests very fragile.
To isolate the system you may use test databases (using sqlite’s :memory: for example) or dummy services instead of mocks.
Use Page Objects
You probably want to test the HTML output. For that you’ll need some selectors such as XPath expressions or CSS selectors to access the page contents. You’ll rely on parts of the DOM structure such as element names, CSS classes or IDs for that. Instead of hard-coding those in every test, create page objects. So then when the page layout changes, you’ll only have to adjust the page objects, not every test.
A very simple way to create a page object is to have one class per page type and define property on that. For example:
from lxml import etree
class Page(object):
"""Page object base class. Instantiated with some HTML."""
def __init__(self, body):
self._body = body
self._doc = etree.fromstring(body)
def __str__(self):
return self._body
def _get_text(self, xpath):
return self._doc.xpath('string(' + xpath + ')').strip()
class DetailPage(Page):
def get_title(self):
return self._get_text('/html/head/title')
def get_page_title(self):
return self._get_text('//h1')
This is just a simple example of how to do this. It expects a valid XML document as input as it uses the lxml parser. You could instead use BeautifulSoup or something similar.
An example for using these page objects:
def test_get_item(self):
"""app is an instance of webtest.TestApp"""
response = app.get('/item/id')
page = DetailPage(response.body)
assert page.get_title() == 'Test page - Memonic'
assert page.get_page_title() == 'Test page'
This assumes an existing app object where the test executes a request on. It then checks that the head title and page title are of the expected value.
Conclusion
Functional tests can mean a lot of satisfaction as they test the actual application. Some of my best success moments are when doing test-driven development with functional tests without ever opening a browser. It really allows me to get in a state where I just focus on what I want to accomplish and stay in a very productive mode.
If you have followed my tutorial so far, you now have a solid testing framework, good unit tests and can now progress with testing the meat of your application with functional tests.
In the next installment I’ll cover testing of the Javascript in your application.
Barcamp Rapperswil
We’ll be talking about mostly geek-topics during the event – though the topics depend entirely on you as a participant. Come and participate. I’ll present as well, though I still have to find a good topic.
You can find all the details and inscribe yourself on the official BarCampRapperswil page.
I hope to see you there.
Unit testing
Introduction
A modern application is split up in many functions, classes or modules. In unit testing you test the individual components and make sure they work as specified. This contrasts with functional or integration testing where you test multiple components at once.
As an example let’s assume you’re writing a web based address book. There will be functions to get a list of addresses from the database, update an address in the database, verify credentials of a user who wants to log in, etc. All of these will be tested with unit tests. A functional test on the other hand would for example test the web frontend by creating an address using the web form and then check if it can by found by the integrated search. So a functional test will rely on many of the base units.
To be able to write good unit tests, testability is an important factor. It basically means that you write your units completely independent of each-other. Testability will be covered in detail in a later post.
Now let’s get started with unit testing. Nowadays the basic unit of abstraction is a class. So I usually write one unit test class for each class in the code.
Example
Again all examples are in Python, but should be easily translated into your programming language of choice.
For starters let’s assume we have this model class which handles database access for addresses:
class AddressModel(object):
"""This is a very incomplete address book model."""
def __init__(self, db):
"""Constructor. db is a sqlite3 object."""
self.db = db
def get_address(self, id):
sql = "SELECT name FROM adr WHERE id=?"
c = self.db.cursor()
c.execute(sql, (id,))
row = c.fetchone()
if row:
return {'name': row[0]}
else:
return None
def setup_db(self):
"""Create the database table."""
self.db.execute("""CREATE TABLE adr (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name VARCHAR(300)
)""")
Forgive the ugly code, but it’s the shortest working example I could come up with.
The database is explicitly passed into the model. This might seem strange but this kind of dependency injection is the best way to ensure testability. Also the model is responsible for creating the database model. This will come in very handy for tests where the best way to test is using an in-memory database.
Now let’s write a simple test case. For this demo I just appended the following code to the same file as the address model (test.py) – though in a real project I would of course separate those.
import unittest
import sqlite3
class TestAddressModel(unittest.TestCase):
def setUp(self):
self.db = sqlite3.connect(':memory:')
self.model = AddressModel(self.db)
self.model.setup_db()
def create_user(self):
"""Create a simple test user."""
self.db.execute("INSERT INTO adr (id, name) VALUES (2, 'Tester')")
def test_get_no_data(self):
"""Database is empty, get_address returns None."""
self.assertEqual(self.model.get_address(1), None)
def test_get_other_id(self):
"""Database has one entry. Requesting a different id returns None."""
self.create_user()
self.assertEqual(self.model.get_address(1), None)
def test_get_name(self):
"""Database has one entry, requesting that entry returns a dict."""
self.create_user()
self.assertEqual(self.model.get_address(2), {'name': 'Tester'})
def test_get_sql_injection(self):
"""Ensure SQL Injection is not possible. See BUG-123 for details."""
self.create_user()
self.assertEqual(self.model.get_address('id --'), None)
if __name__ == '__main__':
unittest.main()
The resulting file can be executed using python: python test.py
Ground rules
There are a few very simple rules I try to follow when writing unit tests and I wrote these tests accordingly.
1. Don’t repeat yourself (DRY)
This is a general coding guidelines, but one that should also apply to tests. Make liberal use of the setUp method that most testing frameworks provide. Also extract common test scenarios (fixtures) into methods or even external data files. The create_user method above is an example.
2. Document the tests
Each of your tests should be documented. Where you have an entry in your bug tracking system, make sure you link to that. This way when the bug pops up again, the next developer has a lot more context.
The docstrings I use here in Python are particularly useful as they get printed when the test fails. For example:
======================================================================
FAIL: Database has one entry, requesting that entry returns a dict.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/pneff/Desktop/test.py", line 50, in test_get_name
self.assertEqual(self.model.get_address(2), {'name': 'Tester1'})
AssertionError: {'name': u'Tester'} != {'name': 'Tester1'}
3. Keep your tests small
Each test method should be as small as possible. If you make them too big it will become very hard to find out what exactly is failing. As a rule of thumb you should call only one method from the unit you’re testing.
4. Isolate tests
Do not rely on the order your tests are executed in. This happens more often than you’d think. For example let’s take this test class:
db = sqlite3.connect(':memory:')
model = AddressModel(db)
model.setup_db()
class TestAddressModelBadlyWritten(unittest.TestCase):
def test_get_empty(self):
"""Database is empty, get_address returns None."""
self.assertEqual(model.get_address(2), None)
def test_get_name(self):
"""Database has one entry, requesting that entry returns a dict."""
db.execute("INSERT INTO adr (id, name) VALUES (2, 'Tester')")
self.assertEqual(model.get_address(2), {'name': 'Tester'})
On my machine this works at the moment. But when I rename test_get_name to test_get_no_data it stops working. So this is a test where isolation failed.
This example might be clear and easy to spot and is also easy to fix (set up the model in the setUp method as was done in the original example). But this problem may sometimes be hidden behind a few layers of abstraction. A few examples of where you can encounter this:
- You may be using a central database (the same sqlite file, MySQL database, etc.) for all tests.
- You may be using some global variables or static class variables in your code.
- A lot of code saves some state on the file system.
- When relying on a web service that service can store central state.
You can often solve those problems with dependency injection and by using mocks objects.
If your tests are properly isolated you can also parallelize them. Try for example the nose multiprocess plugin.
Conclusion
Writing tests for a big web application is not easy. But if you follow a few simple rules and properly isolate the core units you have a good foundation to build on.
On the basis of good unit tests you can then go on and write functional tests.
Testing framework
The initial setup of the testing framework is very crucial. You need to make it as easy and fast as possible for all developers to run the tests. You also need to make sure those tests are executed automatically even when developers become too lazy. That second part I’ll cover in the continuous integration post at a later stage.
In my opinion, setting up a good framework is the most important work you do when it comes to testing. Any work you do here will be multiplied later when you and your team spend less time on writing the tests.
I recommend you start by just adding a tests folder to your project and include one empty feel-good test. See below for a Python example of such a test.
Most modern web frameworks already include a complete framework ready to run. Ruby on Rails started that trend and almost every modern MVC framework comes with a similar testing framework. So chances are that all the hard work has already been done for you. Refer to your framework’s documentation for details.
If you aren’t using a web framework, here are some pointers for getting started outside of that context:
For all of these tools I tried to find a tutorial that shows how to set up a full test, though they aren’t all very complete. If the tutorial doesn’t cover that part yet, spend some additional time to make the tests runnable on the command line with just one command. To achieve that, the Ruby example above uses Rake, nose is already a one-stop testing tool and Java developers commonly use Ant. In PHP projects I have used Rake as well, to run all the tests in a project easily.
If your work mainly in an IDE, you may want to get good integration to execute tests easily as well. I can’t give you good pointers for that, as the configuration is very different between IDEs. But you’ll easily find the required information with any search engine.
The tests I usually include to make sure the framework is up and running are these two:
def test_true():
assert True
def test_false():
assert False
This is Python code to be run with nose. But translating that to your framework won’t be a problem.
If you have good tutorials for other languages and frameworks, please leave them in the comments so I can include them in the article.
Setting up a full stack for web testing
Many developers see the value in getting started with testing their web
applications, but don’t know how to actually start using it. I’ll write about a variety of products and processes which cover the whole range of testing web applications. The posts in this series will be published in the coming weeks. You can find a rough table of contents below but the list may get modified or extended.
Approach
Initially you must spend some thoughts on how you want to approach testing. The main obstacle is that almost always you start with an existing project that already has a big code base. It seems impossible to ever get enough testing in place to be meaningful. But my experience shows that even very little coverage can already improve quality of your system a lot.
I recommend to enforce two very simple rules:
- No bugfix without a test.
- Develop new features using the test-driven methodology.
But whatever happens, don’t go and try to implement 100% code coverage for your legacy code. That will kill you – and drive motivation down very quickly.
Implementing a test for each bugfix is very little work. To fix the bug, you have to reproduce the problem anyway. During that process you will find at least one test case that can be implemented in an automated fashion relatively easy.
The first few tests will be the hardest. But don’t be discouraged by that, it will get easier. It’s hard mainly for two reasons. First and most important, testing is something you have to learn. You didn’t learn your programming language in just a few minutes either. Second a lot of software is not written with testability in mind.
I hope this series will help you getting better with both these aspects.
Table of Contents
This table of contents will get updated when I write the articles. Keep coming back to it.
- Introduction
- Testing framework
- Unit testing
- Functional testing
- Javascript testing
- Continuous integration
- Test-driven development
- Testability
- Measuring coverage
Semantic Data
Graph databases are basically one of the many different ways to store data – something which is currently being hyped under the awful name “NoSQL”. So much so that Neo4J, one of the well-known graph databases, even uses that term in their title now.
I’ve recently seen two talks about this. First there was the great talk Why Semantics? by Toby Segaran author of the book Programming Collective Intelligence. Then at yesterday’s swiss.py meeting Cathrin Weiss presented her Hexastore work which she implemented in Python.
If time permits I’d really like to play with Neo4J or openRDF Sesame. There is at least one project on my back burner where a graph database would probably make a lot of sense.
Toby has also just published the book Programming the Semantic Web though I haven’t read it yet.