Testing Testing

Since I claim to advocate Agile development practices, I recently decided to give unit testing, the writing of simple test suites that exercise an application's basic functionality, a solid try.

I've been in various organizations that have tried to do it, but the effort always founders and fails. Everyone knows unit tests are a good idea; no one ever gets any in place. When the subject comes up at job interviews, other companies seem to have had the same problem. New features always creep ahead in priority, and once the code becomes too big, writing unit tests becomes too daunting a task. Plus, the new feature is working, more or less, so the motivation decreases to write tests for it.

As I started my Mac/iPhone programming, however, I realized I had the perfect environment for unit testing: no deadline and no legacy code. So I wrote new code and the test cases to exercise it at the same time. (I didn't do test-driven development, where you write the tests first and then write the code to make them work.)

I haven't since become a unit testing zealot — at least I don't think so — but I have a renewed enthusiasm. In a normal development scenario, I would have written all my support classes and then a UI so that I could test them. If there was a bug, I'd have to step through all the layers I had written to find it. With unit testing, I found several bugs long before I had a UI. When I did attach a UI, the fact that it worked smoothly out of the gate was anticlimactic: I knew the underlying structure was solid, after all.

Recently, I thought of some code reorganization I could do on the project as well as some significant performance improvements. I spent an hour or so doing the work, shuffling this here and that there. I set up my unit tests to run, and they passed: I could assert that my changes hadn't broken anything major. That's a really good feeling; there was a bit of a rush when I saw "Build Succeeded" flicker by in the log.

Even with my short experiment, I realized the two major benefits of unit testing: catching bugs earlier ("The earlier you catch bugs, the cheaper they are to fix," runs one programmer mantra) and making sure that major code changes don't affect the core functionality.

But how to bring that to work? At Maxis, we have a lot of pressure to deliver features, an always-busy team, and a big ball of legacy code even for the website. Just like every other tech company. One of my co-workers and I have talked about it, and the only answer seems to be: just start writing them, and keep them running. The first few will be tough, because you need a support structure for them (especially in the form of mock objects, layers that conform to the contract of the real layer, but deliver static, simplistic data), but once they get going, hopefully it becomes easier to add new ones. I have a squirrely bit of logic in one piece of code: That's my first candidate.

The problem is selling the concept to the business side. I think in a lot of companies, the extra time spent writing unit tests is seen as time that could be spent writing a new feature. In a way that's true, but I now see that unit tests actually are a feature: stability. No one wants an unstable system, but the investment in stability doesn't obviously pay off for a long time. In fact, if you've done it right, the investment never seems to pay off because the system doesn't have as many problems. You can't measure how many problems there would have been if the investment hadn't been made. But you can measure the number of users using a new feature. So you get instant feedback that it was a good addition, and you provide a metric. How does unit testing compete against that?