Code Coverage Metrics and Testing

For the last few months, I've been tracking some metrics about Tapestry. On each release, and occasionally in between, I've been tracking the NCLOC (non-comment lines of code), the number of Java source files, the number of tests, and the percentage of code coverage (according to Clover).

I have an almost religous belief that you must have at least 85% code coverage for a project; I think the bugs "hide like cockroaches" and are only flushed out when you hit that magic number. Thus, I spend most of my coding time writing tests, not production code. The trick to staying sane is to remember that production code and test code are all just code, of the same cloth, each useless without the other.

You can see the amount of code growing with occasional downward spikes as dead code is located and eliminated (the large spike around 8/10 represents a change in how I was counting code, counting the code for each sub-project seperately -- these charts are just for the framework itself).

And here you can see the stuggle to keep the code coverage about 85%. Over this same time range, the number of unit tests has risen from 906 to 1,257.

In fact, the news is actually pretty good; in Tapestry most new code, code introduced for release 4.0, has close to 100% code coverage. These charts are just for the framework proper, but the annotations library, for example, has true 100% code coverage (every line of code, every branch).

The majority of new tests are written as true unit tests, on top of EasyMock ... all but about 40 of the tests, which execute mini-applications inside a simulated servlet container. Those 40 tests take about 220 seconds to execute; all the remaining tests combined take about 25 seconds. Thus the power of unit testing.

All told, the Tapestry project includes over 1400 tests. And Tapestry is built upon HiveMind, which has about 740 tests itself.

Personally, I'd like to see the code coverage numbers jump up into the 95% range. I think Tapestry 4.1 will start with a purge of dead and deprecated code, and then a lot of filling in the gaps in the unit tests, before we start working on new ideas. And I have a dream of simplifying the integration tests and productizing them, so that they can be used by Tapestry developers to test their applications. What would be great would be to link together something like jWebUnit with a mock-up of the servlet API, for true unit testing (without having to run the application in a seperate process).