Testing Large Code Bases

Keith Stobie's article on testing large systems is a good read, but only if you're pulling out bits and pieces of the article. I thought the piece was a bit rambling and without a coherent thesis, but there's good coverage on several topics all the same. Stobie seems to somewhat throw up his hands regarding "handcrafted" tests for large systems as being too cumbersome. He uses an example of trying to test square root values for 32- and 64-bit machines as being completely unreasonable in that it would take 584 years to run at one nanosecond per test. Much of the rest of the article stuck me in the same bent: testing methodology "X" won't work for large systems because of reason "Y". That's fine and might even be true, but I don't think one tool works for every situation. Constructing a solid system requires a well thought out test plan specifically addressing test requirements at all stages. Assertions ought to handle specific situations in the code itself. (See Maguire's book.) Unit tests need to hit boundary checks as well as input verification and 100% branch testing. They should also do some sanity checks by blasting the system with very large over- or undervalue situations. Integration tests need to be well-designed to check component interractions and verify functionality boundaries. (See Kernighan and Pike.) Acceptance testing needs to be tied closely with requirement verification. Finally, any large system needs a couple SOLID testers who do nothing more than random bashing of the system in attempts to break it. (See Whittaker.) "Users wouldn't ever do that particular action!" Horse puckey, and deep down you know it. I did like Stobie's point on code coverage != quality. Fixating on coverage buys you nothing if you haven't first laid out a solid testing strategy (boundaries, branches, etc.), or if you've badly implemented those tests. Code coverage will point out to you places you've missed implementing tests for. Code coverage is a tool, not a panacea. More fundimentally, a well-tested system uses all of these various methodologies. It's even more important as a system gets larger.