Testing the Tests via Mutation
Wednesday, October 29th, 2008 by Max
To set the ground for this post I first need to reiterate a message that has been heard from many other sources. Code coverage can only tell you how bad your test suite is, not how good it is. Meaning that if you have bad coverage you know something is wrong, but a high coverage number tells you virtually nothing. This message seems obvious for many developers out there, but it’s not heard by a surprisingly large section of them.
I have been a teaching assistant for several undergraduate and graduate testing classes. Every year when I grade their unit testing projects I always get several of them (about a quarter) that hand in tests with “good” coverage, but not a single assert statement to check the results of their tests. This is even after I get up in front of the class and rant about the exact problem from previous years. Needless to say those students tend to do poorly.
So, if test coverage is not a good metric, how can developers measure the quality of their test suites? I’m going to talk about one approach in this post, mutation testing.