Patrice's Weblog

I recently spent a long day and night debugging a SAML login issue over at Squirro. Eventually we found out that the xmlsec version shipped with RedHat Enterprise Linux 8 is broken for signing.

Error Introduction

As part of our Single Sign-On implementation we use pysaml2. That library in turns depends on xmlsec, a library supporting XML signature and encryption. In the deployment we were debugging we saw that the xmlsec1 call was erroring out:

$ /usr/bin/xmlsec1 --sign --privkey-pem privkey_file --id-attr:ID urn:oasis:names:tc:SAML:2.0:protocol:AuthnRequest --node-id id-qu56UdhaX0ojT90eL --output /tmp/tmpg48_9_ls.xml /tmp/tmpqusxu0my.xml
func=xmlSecOpenSSLEvpSignatureExecute:file=evp_signatures.c:line=466:obj=rsa-sha1:subj=EVP_SignFinal:error=4:crypto library function failed:openssl error: 67526835: rsa routines: rsa_ossl_private_encrypt missing private key
func=xmlSecTransformDefaultPushBin:file=transforms.c:line=1921:obj=rsa-sha1:subj=xmlSecTransformExecute:error=1:xmlsec library function failed:final=1
func=xmlSecTransformIOBufferClose:file=transforms.c:line=2547:obj=rsa-sha1:subj=xmlSecTransformPushBin:error=1:xmlsec library function failed:
func=xmlSecTransformC14NPushXml:file=c14n.c:line=243:obj=exc-c14n:subj=xmlOutputBufferClose:error=5:libxml2 library function failed:xml error: 0: NULL
func=xmlSecTransformCtxXmlExecute:file=transforms.c:line=1037:obj=exc-c14n:subj=xmlSecTransformPushXml:error=1:xmlsec library function failed:
func=xmlSecDSigCtxProcessSignatureNode:file=xmldsig.c:line=550:obj=unknown:subj=xmlSecTransformCtxXmlExecute:error=1:xmlsec library function failed:
func=xmlSecDSigCtxSign:file=xmldsig.c:line=286:obj=unknown:subj=xmlSecDSigCtxSignatureProcessNode:error=1:xmlsec library function failed:
Error: signature failed
Error: failed to sign file "/tmp/tmpqusxu0my.xml"

Summary of Findings

What we eventually found out is that the xmlsec package version 1.2.25, which is shipped with RedHat Enterprise Linux 8, has a bug when signing documents. The bug manifests itself only when the private key being used has a corresponding public key in the system’s certificate registry (usually /etc/pki/tls/cert.pem).

The bug was fixed as part of the 1.2.26 package. The actual patch is a one-line change.

Fixes

There are currently two known fixes for this problem:

1. Install an updated version. Unfortunately this is currently not readily available in RedHat Enterprise Linux 8, so a patched package has to be built manually.
2. Remove the matching public keys from the local certificate file. This will result in error messages (“certificate verification failed”) but the signing succeeds regardless.

Investigation Details

This section includes some information that has already been included above,
but in somewhat more detail.

When signing, an xmlsec1 command is executed as follows:

/usr/bin/xmlsec1 --sign --privkey-pem privkey_file --id-attr:ID urn:oasis:names:tc:SAML:2.0:protocol:AuthnRequest --node-id id-qu56UdhaX0ojT90eL --output /tmp/tmpg48_9_ls.xml /tmp/tmpqusxu0my.xml

In our initial investigation we thought this could be due to some security hardening around crypto policies.
This was supported by the problem not appearing when testing in a different environment using Rocky 8.
However setting the crypto policies to LEGACY did not solve the issue.
Additionally upon further debugging we were able to reproduce the issue also on Rocky 8 without any hardening having been applied.

The next clue was that the problem did not appear when the corresponding public key for the provided private key could not be found.
In that case the following error would appear but the signing actually took place (some internal information redacted):

func=xmlSecOpenSSLX509StoreVerify:file=x509vfy.c:line=341:obj=x509-store:subj=unknown:error=71:certificate verification failed:X509_verify_cert: subject=…; issuer=…; err=20; msg=unable to get local issuer certificate
func=xmlSecOpenSSLX509StoreVerify:file=x509vfy.c:line=380:obj=x509-store:subj=unknown:error=71:certificate verification failed:subject=…; issuer=…; err=20; msg=unable to get local issuer certificate

These combined findings allowed me to debug the problem better by reviewing the xmlsec and OpenSSL source codes (versions 1.2.25 and 1.1.1k as used in RedHat Enterprise Linux 8).
Not having previously been exposed to the source code of either of those libraries, this made for an interesting experience.
The problem was eventually pinpointed to an xmlsec function called xmlSecOpenSSLKeyDataX509VerifyAndExtractKey which seems to assign a public certificate into a structure which later expects a private key.

Armed with this information, I then tested other versions of the library and quickly found that xmlsec 1.2.26 was no longer affected by this problem.
A few steps of git bisect later I finally found the patch for this problem, which had been authored two days after the release of 1.2.25.

“select” isn’t broken – except when it is

As a debugging project this one was a lot of fun. Throughout my work I have always kept this Pragmatic Programmer rule in mind:

Remember, if you see hoof prints, think horses not zebras. The OS is probably not broken. And select is probably just fine.

I always assume my own code to be the culprit of any bug until proven otherwise. In this particular instance it turned out different for once. I could definitely have saved a lot of time by testing other xmlsec versions earlier, rather than narrowing in on the exact dependencies I was dealing with. And initially I was a bit upset with myself for not doing exactly this.

But upon further reflection I had to remind myself that this mindset has certainly saved me much more debugging time in all the instances where it was indeed my code that was broken.

TL;DR: I’m shutting down list.blogug.ch and stats.blogug.ch, unless you save it by taking over.

Over the past few years I have been maintaining parts of the blogug.ch project, namely the list and the stats. I haven’t done any actual development on those since 2006 but they have been running fine.

From time to time I even confirmed the blogs to make sure they match our community guidelines. But time has been short for that and the list of unconfirmed blogs has been constantly growing with nobody else picking up the slack. This even though a few people have admin access and access is liberally granted. And of course the stats project shows that people just aren’t blogging anymore.

The final straw is that I now had to switch servers once again, as the old hoster proved unreliable.

So all of this combined brings me to the following decision: I will not migrate my blogug.ch projects to the new server and will no longer maintain them.

Please contact me if you want to take over. I’ll give you the full source code and access to the server.

To stay updated on all things geek I have been relying a lot on Hacker News lately. But not by using the main web site which would steal too much time from the work I should be doing. Instead I use two great services, one for daily and one for monthly updates.

Hacker News daily is a blog where each day the top 10 Hacker News stories are posted. The stories are deduplicated, so only new information is shown. Using Google’s Feedburner I converted this into a newsletter. Feel free to subscribe to the resulting Hacker News Daily Newsletter.

Then there is Hacker Monthly. It’s a magazine for which you need to pay a subscription fee. You get a beautiful e-magazine once a month with curated stories that were on Hacker News in the previous month. This is especially handy for the Kindle to which the newest edition gets delivered automatically.

I have a simple password policy: every account gets its own password. To manage this I store them in a password manager.

When switching to Windows one of the most painful things was replacing that password manager for which I had used the Mac Keychain. It’s a password storage, secured with either the login or a separate password. It integrates extremely well into the system. Apple even ships pre-installed Subversion clients and SSH agents that store their passwords in the Keychain. And for non-integrated applications, getting a password out is relatively easy using a simple keyboard shortcut. That is topped off with secure notes for credit card information and similar.

Since Windows XP Microsoft now ships a Credential Manager. But when comparing it to the Mac Keychain it fares very poorly. It doesn’t have a flexible way to enter your own passwords, doesn’t have a search, nor an option to manually get access to a password again and — worst of all — no program except the Windows Explorer and VPN software seem to integrate with it. Not even Internet Explorer. So I looked for a good password manager on Windows. I’m still not comfortable with cloud-based password managers such as Passpack or LastPass (which might actually have had a security breach a few months ago).

I ended up with KeePass, an open source software for storing passwords and other sensitive information.

Using it is very simple. When creating a password I enter a title and the username I chose on the site. Then I copy out the password that was automatically generated and paste it into the web site signup form.

For accounts that have some sensitive data in it I set an expiry date, usually one year in the future. When opening KeePass it shows a list with all expired entries so I can go and change those passwords.

When it comes to logging in on that web site again, I use the Auto-Type feature. So I navigate to the web site and press the global hot key (defaulting to Ctrl+Alt+A). This will find the right user name and password and enter it in the currently active window. If I have multiple accounts on a site as often happens for business vs. private accounts, KeePass presents me with a small prompt to ask me for which account I’d like to use. Auto-Type can also be heavily customized to work for web sites that don’t have a typical username and password login form.

On the iPhone I have access to the same password database using MiniKeePass.

So overall I’m pretty happy with KeePass. But there are some web sites where the default Auto-Type doesn’t work well. One culprit is Google. For the initial login they ask for the username and password. Then a few days later — for security reasons — they ask for just the password. But in those cases quickly searching for the password in KeePass and copying it using a keyboard shortcut works well enough.

A few months ago I switched from Mac to Windows. The reason was work. At Memonic we develop a consumer application which should work well for non-geeks and average consumers. But a while ago I noticed that at our office every single one of us was either on Mac or Linux. While you can test Windows apps well enough using VMWare that just didn’t happen in practive — not least because of the heavy memory tax of running Mac and Windows side to side.

So I decided to switch to Windows full time. Not only that, I vowed to make Internet Explorer my main browser. That of course means that my IQ is below average (or not).

The switch wasn’t nearly as painful as I had expected. Here’s a list of tools I need to make it worth the while.

• TotalCommander — TotalCommander is a magical file browser. It’s the very first thing I installed and the app I missed most on Mac and Linux.
• Launchy — An application and document launcher like Mac’s Quicksilver. I’m not 100% happy with Launchy and from time to time look at alternatives.
• Vim — Finding a good TextMate replacement wasn’t easy. With most editors I tried it came down to project support that just wasn’t to my style. If I remember correctly that’s also why I ditched E. But with Vim and the Project plugin I’m happy for for now. The individual vimprojects files are index by Launchy so I can easily open a project.
• PuTTY — Every server I use is set up as a bookmark and indexed by Launchy.
• Console — A proper console for Windows. Using Tabs you can run Windows’ own shell, Cygwin bash or the Powershell. I even have tabs for various Python versions and for a tail command I use often.
• VirtualBox — For those times I need an old Windows version or want to try something on Linux I use VirtualBox.

If there is any important app that I should be checking out, please tell me.

Over on the Memonic blog I blogged about why and how we work with Python at Memonic.

The blog post goes into the architecture we run here and how Pylons and WsgiService fit into that. I hope you enjoy the read.

I spent the past weekend in Paris and tried to go paperless with the iPhone. At the same time I wanted to keep data roaming deactivated. So before the trip I researched some offline applications to make the best out of this vacation. And was very surprised by the selection of very good apps that do not need permanent Internet access.

I’ll quickly introduce the apps I used.

OffMaps

OffMaps is an iPhone map application using the OpenStreetMap data. That data is now surprisingly complete for many European cities and even includes many POIs such as restaurants, bars, public baths and many more.

OffMaps compiles that POI data into easy to use guides that also allow for offline search. Also you can download the full map of a guide or any area you choose to use it offline.

For the guides you need to pay extra. You can either buy single guides or – as I did – just buy the flat rate so you have great local information wherever you go.

The only information I missed was restaurant recommendations instead of only the locations. And routing does not work offline.

Memonic

Memonic is the online tool to take notes and save web site clippings I co-founded. Before the trip I saved the hotel reservation and some travel tips to Memonic.

The Memonic iPhone application downloads the whole collection for offline use. So in Paris I had access to all my Items without trouble.

Métro

All our traveling in Paris was either on foot or by public transport. Métro is a generic public transport app with data for many cities. Among others of course Paris. Once the data is downloaded you can get routes offline. I didn’t get to like the user interface during my few days of working with it but it always gave me correct answers very quickly.

Lonely Planet Paris City Guide

When the Eyjafjallajökull came upon us, Loney Planet decided to make some of their city guides available for free for some days. So I downloaded their Paris guide and used it in Paris.

In summary I didn’t like it at all. For starts, the information was too short, contained almost no links between different places and many copy-editing errors such as completely duplicated paragraphs made it into the product. But the real killer was lack of access. The app relies too much on search and doesn’t offer a good browsing access to find interesting places.

It does contain some restaurant recommendations and I did try one of those and liked it. But I ended up relying more on OffMaps even for that task.

Stanza

Stanza is an E-Book reader for the iPhone. It provided me with reading material for the train rides.

Built-ins

I also relied heavily on built-in applications. Calendar for storing itinerary information, Contacts for knowing where to send postcards to, Notes for some on-the-road information and the Clock for waking up in the morning.

Next week on February 18 we’ll have the first meeting of the Swiss Puppet User Group (SPUG). The meeting is in Bern at the offices of Puzzle.

The first time we’ll have some lightning talks. I’ll do a presentation about my personal Puppet setup on my Mac workstations.

Get all the details on the official SPUG web site.

This post about JavaScript testing is part of the Setting up a full stack for web testing series.

Introduction

JavaScript tests are sufficiently special to deserve a post outside of unit and functional testing. You need a special set of tools to write these tests. But then again for JavaScript the usual split between unit and functional tests is still applicable. So what I wrote about those two topics is also relevant here.

In this post I cover two separate tools for JavaScript testing: JsTestDriver, which is great for JavaScript unit tests, and Selenium which can be used to functional test a web application that uses JavaScript.

Selenium

Selenium is a software suite used to remotely control browsers. It consists of various sub-projects:

• Selenium Core: This is the original Selenium. It allows you to create a HTML page with some browser commands. Those commands can then be run in the browser when you open the page.
• Selenium IDE: A Firefox extension that can record your actions in the browser. This is a great way to learn the Selenium syntax as you can then export the recorded actions in several programming languages.
• Selenium Remote Control (Selenium RC): A Java server that allows controlling browsers remotely. Can be used from any programming language using a simple HTTP API.
• Selenium Grid: Basically the same as Selenium Remote Control but for running in parallel.

Personally I only use Selenium RC, so I won’t talk about the other parts. To get started, read the excellent Selenium documentation which explains everything a lot better than I could. If you want a quick start I recommend the Selenium RC chapter.

The basic test case in Selenium works like this:

1. Open the page to test (with by first logging in)
2. Execute some action like a click on a link or a mouse event
3. Wait for the action to execute (Ajax loads to finish, etc.)
4. Assert that the result is as expected

These four steps are repeated for every test case – so a lot of pages will be opened. Opening all those pages is the reason why Selenium tests tend to be very slow.

As a test framework you can use whatever you already have – in your favorite programming language. The difference is just that in your tests you will talk to Selenium to get the current state.

Take the following Python code as an example:

def test_login():
    sel = selenium("localhost", 4444, "*firefox", "http://localhost:5000/")
    sel.start()
    sel.open("/login")
    assert sel.is_text_present("Login")
    assert sel.is_element_present("username")
    assert sel.is_element_present("password")
    assert sel.get_value("username") == 'Username'

This script launches a Firefox browser and opens a login page of an application running on the localhost. It then gets several values from Selenium and asserts the correctness of these values using the standard test framework methods.

JsTestDriver

JsTestDriver is a relatively new tool which can be used to submit tests suites to browsers. Those browsers have to register with a Java-based server and you execute tests by submitting them to that same server.

So far that sounds very similar to Selenium. The difference is that JsTestDriver works with a blank page in which it directly inserts the test suite. It does that with a lot of optimizations to make sure the test runs are as fast as possible.

After that the unit test suite is run – directly inside the browser – and the client gets the test results including stack traces if available.

I recommend the Getting started documentation on the JsTestDriver wiki to see some code.

One of the main differences to Selenium is that you write the tests directly in JavaScript. There is a built-in test framework but you can also use other frameworks depending on your taste.

To show some code that you can contrast with the Selenium example above, consider this example:

GreeterTest = TestCase("GreeterTest");

GreeterTest.prototype.testGreet = function() {
  var greeter = new myapp.Greeter();
  assertEquals(“Hello World!”, greeter.greet(“World”));
};

Differences

Selenium is a very magic piece of software. Everybody falls in love with it when seeing their browser doing all the work. It’s something people just can’t resist. And so they end up using Selenium for all their web testing needs. I’ve been there myself.

The downside of Selenium is that it’s very brittle and slow. This is something that can’t really be avoided because all it does is control a browser. Opening pages and waiting for all scripts to load takes some time. And doing that hundreds or thousands of times, as is easily the case in a large test suite, leads to very slow test executions.

So instead of falling into that trap and only using Selenium, I recommend to clearly separate out unit tests which you can then execute in JsTestDriver. JsTestDriver does a lot less work and because of that is a lot more stable and faster. Then do integration tests with Selenium to test some basic workflows.

As an example take an autocompletion widget which is used on a search home page. Almost everything the widget does you can test by just calling its public interfaces and seeing if it does the right thing. This includes all the strange edge cases such as handling network timeouts or invalid data. So this part you do with a big JsTestDriver test suite. Then you only need one small functional test case to make sure the widget is correctly embedded in your home page. That’s your Selenium test.

As is evident I’m very happy that JsTestDriver has come along. Before that the only good solution for JavaScript testing was Selenium – and as I explained above it’s not a perfect fit for every testing need.

Conclusion

If you have followed my testing tutorial so far you now have all your testing tools set up. Some more chapters will follow but those now cover testing philosophy and tools around the principal testing framework.

The past few days I created a mobile site for Memonic. During this I made use of some of the newly found knowledge from the past Webtuesday which Adrian Kosmaczewski was kind enough to share. See also my notes about iPhone web development from that event.

The goal of this mobile site was to get a broad device coverage. As we’ll create an application for the iPhone, it was especially important to get a nice site for all the other devices. During his talk Adrian mentioned the Yahoo! Blueprint framework and I was sold quickly.

The basic architecture is relatively simple:

The really nice thing is that as a developer you don’t have to worry about any of the device detection, HTML generation, CSS fiddling, etc. That’s all handled in the “Yahoo server” box.

For our internal evaluation I gathered the following list of advantages and disadvantages:

Advantages:

• Very broad device coverage
• Fast development
• Image compression done by Yahoo
• Less code

Disadvantages:

• Mobile frontend hosted by Yahoo without any SLA (but transparent to the users, we can still serve it using the m.memonic.com domain)
• No HTML (we have to convert HTML to XForms XML – that’s a problem for detail pages on Memonic)
• No custom design (though to some extent that’s coming next year)
• No integration in Google Analytics (but we probably can use Yahoo’s own statistics tool for the mobile part)
• Currently no price plan

Based on that list we decided to go with Blueprint for the moment. The architecture means, that whenever we have to migrate away we can probably re-use our existing code and build something similar to Yahoo! Blueprint ourselves. That would actually be a really nice open source project.

For now: thank you Yahoo! for offering a wonderful service.