Case Studies on Benefits of TDD

I’m a believer in test driven development. For many reasons. I don’t do it enough myself, the people I’m around don’t do it enough, the people I talk to don’t do it enough. For many reasons.

One commonly cited reason is a lack of evidence about TDD’s benefits. I’ve always found that a bogus argument, because I think it’s fairly easy to look to specific metrics around code complexity and bugs in a before/after environment.

Without further ado, here are some links pointing to various studies on TDD. Note these links are to TDD. I didn’t even go down the route of “show me stuff that backs up the benefits of unit testing, not just TDD.”

Using Bing I Googled this search phrase: “test driven development studies” and got these great hits:

From Biblio, an entire page of studies.

In InfoQ, an article on a study from folks at Microsoft and IBM. The InfoQ article links to the Empirical Software Engineering Journal’s article which costs $34 to read. Evil, greedy, profit-grubbing Microsoft hosts the article here for free. A comment in the InfoQ article points out some great contrarian questions which you need to ask yourself about TDD.

There’s also a video on Channel 9 interviewing one of the study’s authors. (I haven’t watched it. Yet.)

An interesting paper summarizing several other studies in industrial, semi-industrial, and academic environments. Benefits in the latter two environments were not as apparent, particularly in the academic. I wonder if some the low benefit in the academic environment is due to a lack of real-world experience both in the students and their instructors.

Does Test Driven Development Really Improve Software Design Quality an awesome article published in IEEE Software back in March/April 2008. Read closely the sections on misconceptions and cons. Chase down some of the other referenced reading there, too.

Hopefully you’ll find these articles and links a good starting point for your own reading. Use them wisely when trying to make the case for selling TDD (or even just unit testing, period) to your teams, management, and customers. Keep in mind these critical points:

• Be honest. Unit testing and TDD aren’t silver bullets. Crap code driven by crap tests driven by crap requirements is still a pile of crap.
• Be honest. Your perceived productivity will drop. You’ll be knocking out less features on any given iteration/cycle/release.
• Be honest. You’ll suffer additional productivity losses as you learn how to do testing, preferably in TDD fashion.
• Be honest. There will be culture and skills issues with your team, your management, your customer, and you.
• Be honest. Testing and TDD is often more about long-term gains than short term. You’re trading feature velocity for overall quality and maintainability.
• Be positive. The gains you’ll realize from moving to TDD are significant: reliability, quality, maintainability.
• Be positive. The speed at which you’ll realize those gains will greatly improve as you move forward and get up over the learning curve.
• Be positive. There’s a growing mountain of solid study-based evidence around the benefits of unit testing, particularly TDD. I emphasized “evidence” because too often we geeks get so fired up about things that we are passionate about that we ignore arguments based on facts and evidence.
• Be positive. There are specific metrics you can use to show the gains: complexity metrics (NDepend, FTW!), code coverage, decreasing bug counts, fewer escaped-to-customer bugs, increased velocity around future maintenance and expansion.

Now go forth and do battle! “Sancho, my armor!”