Over on the Memonic blog I blogged about why and how we work with Python at Memonic.
As the original blog is now archived, I later reproduced the article below.
Python
Fun fact: Memonic is written entirely in Python. Why did we choose this particular programming language? After all - aren’t all the serious developers on Java and the cool ones on Ruby?
When we thought about the programming language to use, we already knew how our architecture was going to look, namely that we’d have a frontend and lots of small backend web services. That’s a setup we had previously implemented successfully at local.ch. But at local.ch we had the frontend in PHP and the backend in Java. At Memonic we wanted everything to run on one language so every developer could feasibly work on the whole stack. This threw out PHP which isn’t so great for daemons and services (but I still like it for frontend work, especially with Okapi). We then had Ruby, Java and Python in the race. Without going into detail about why the other two languages suck ;-) Python won us over because of it’s maturity and the very wide selection of frameworks and libraries.
The main Python components of our setup are Pylons, WsgiService and httplib2. These all warrant an explanation about why we chose them.
Pylons
The sweet spot of most MVC frameworks (in any language) is giving the user controlled read/write access to a usually relatively simple database. We don’t fit into that sweet spot as the frontend does not have any database access directly. Instead our frontend accepts requests by the users, executes some requests to various backend services and then outputs the combined result as HTML. This makes many MVC frameworks relatively painful to use but Pylons works very well for us. While it’s also optimized for the normal database access mode it’s very easy to swap out every aspect of the framework. In my view, this makes it ideal for a project that we hope is going to live a long time. Fingers crossed!
WsgiService
All our backend services represent either a very simple data model or an algorithm. We currently have about 40 of those services. Our main interest was to be able to write them easily without much development overhead. As we really love HTTP we were not particularly interested in any RPC frameworks such as Thrift or Protocol Buffers. Instead we wrote a small framework which makes it extremely easy to write well-behaved REST services in Python: WsgiService. With it creating a new service is so simple that one of our developers actually wrote TWO new services within the first month of starting work here. Which is not to diminish the fact that he is supremely talented!
httplib2
It’s not only important that we are able to write REST services easily, we also need to consume them. For that we use httplib2 which is a great HTTP client for Python. We were especially interested because of the support for caching and the very easy API. We have written a very small wrapper on top of it that mostly deals with directly decoding the JSON responses into Python structures and setting some request headers that should always be present in our environment.
Standing on the shoulder of giants
In addition we use a great many other libraries. It would be too much to talk about all of them. So standing in for about two dozen other packages, I’ll give an honorable mention to premailer written by Peter Bengtsson. This small library of less than 250 lines of code makes the HTML mails we send out rock solid. The fact that this library was available saved us probably a day or two of work and lots more of debug pain. While this may not seem like much, it does add up. Many gems like that (sorry, I mean eggs of course…) can be found in the Python Package Index (PyPI).
Our contributions
We don’t want to just consume code, we have also published some of our own. Currently you can find two of our packages on PyPI: WsgiService and TinfoilHat. WsgiService has been extensively described above. TinfoilHat is a small subclass of httplib2 which is more paranoid about input. For example it does not accept any URLs from private IP ranges. We use that to make sure that an attacker can’t fool our servers into downloading data from a private resource.
In conclusion: we don’t regret going with Python. On the contrary; we have come to respect this language more over time. The way the move to Python 3 is being handled by the community shows that Python also has a future.
If you’re interested in more, I regularly publish web clippings about Python on Memonic.
If this sounds like a technology environment that you would like to work in, see our jobs listings and you may soon join the team. We are a startup located in Switzerland, right in the center of Europe. And we’ve got our hand on the pulse!