31 Days of Testing: Day 2: Setting Expectations

Updated: Index to all posts in this series is here!

Updated again: Fixed some really ugly formatting. Sorry, folks!

We folks in the software industry talk a lot about the importance around setting expectations as we move through a software project. Too often those expectations are only discussed in the context of the system we’re delivering; setting expectations in the context of testing that system too often gets left at the sidelines. (I’d make a vehement argument that you shouldn’t be separating those contexts, but then this post would take me a year to write. Go with me on this, ok?)

Projects with unclear expectations are at an extremely high risk of failure. Take the same care of setting expectations around your testing efforts as you do with expectations of the system itself. (Better yet, have those as one combined discussion—because that’s really how it should be!)

What’s the goal of testing?

No, this isn’t a silly question. Putting the question of “What’s our goal with testing?” out on the table really helps get the team, including management, gelled around who/why/how much.

Here are a few different goals I came up with. It’s not a comprehensive list, but it’s a conversation starter:

Heal customer relationships/Improve perception of company

It happens. Your team/organization/company has done a poor job and you’re now paying the price with irate customers, bad press, and dropping revenues. Yes, poor quality does pay off in the negative sense over time. Now you’ve got to pay the piper. If you’re at this spot it’s not necessarily a bad thing. Someone high up in your food chain has likely figured out the damage that’s been done and is giving some support for changing things. Look at it in a positive light, although you’ve certainly got lots of work ahead of you!

Cut cost of maintenance?

Bugs kill. They’re costly to fix after they escape into production. Preventing bugs from ever being written is the best option. Catching them early in your project is nearly as good, and loads better than fixing after you release.

Identify risk

Zero bugs is an admirable goal, but it’s simply not possible for many teams. (Yes, there’s a sad note of surrender there.) Perhaps your team’s goal of testing is to at least identify for your product owners/stakeholders the level of risk for the software’s current state. At the end of the day the business side of the house needs to be able to make a business decision of “Can we ship this product as is, or do we need more work?” Testing helps identify and clarify the risk.

Fakery: Appearance of quality

Ouch. Yes, this happens. I’ve been in these environments and have left them. Poor companies/organizations use their testing team as a scapegoat or pointless mascot for their awful environments and development practices. “Yep, we’re solid with quality! We’ve got two testers for our 30 developers, and we’re committed to quality!” There’s just no nice way of putting it. Sorry.

Building great software

The best goal is that organizations want testing as an integrated part of their development processes because they simply want to ship great software that works and helps customers solve their problems.

Do an inception deck so everyone’s clear on the goals!

The awesome book Agile Samurai has a tremendous section in it on creating an Inception Deck. I highly encourage you to buy a copy of the book and have a look through that section.

The Inception Deck is a byproduct of a one or two day long workshop where the team lays out clear goals for the project, among other things. You end up with a great concise set of slides which clearly state goals for your project.

These sorts of goals need to be up in your team’s work areas, because these can greatly help keep everyone reminded of where testing fits in the picture, and what the overall goals of the project are. That’s a great head nod to the level of commitment you’ll need for a successful integrated testing effort.

One of the best points in Agile Samurai: Post the deck on a wall in the team area so everyone can see it!

Once you’ve done that, make sure you, your team, and your stakeholders/customers are clear on the many things you’ll need for a successful project. This list is of course focused on the testing-related things!

You’ll need time

Yes, testing requires time. Surprise! You’ll need time for a great many things relating to testing. Time to learn new skills and get through proficiency to mastery. Time to fit your testing into your existing processes, or to create new processes. Time to evaluate and update those processes. Time to learn new tooling you may be making use of. (Yes, even Test Studio, which I’m immensely proud to be a part of, requires time to master.) Time to set up that tooling and infrastructure.

Most of all you’ll need time for doing the work! Testing does add up front time, be it some form of developer-level testing (unit, integration, etc.), or the tester’s final checks. More time up front, dramatically less time after you’ve released. Sometimes that is a tough sell to folks who are too focused on the immediate timeframe.

You’ll need money

You have to set expectations around the cost of testing. Yes, there’s a cost, and it’s not trivial if all you’re focused on is what you’re paying today instead of focusing on the overall cost of the project. You’ll have increased headcount as you hire on new testers to your team. You’ll have increased costs for more infrastructure—that overloaded build server isn’t the right place to be hosting those long-running, intensive integration and functional tests—and you’ll need additional systems for those new team members.

There are costs for new tools, too, either explicitly if you’re buying something like Test Studio, or implicitly if you’re using one of the many great open source tools. (By “implicit cost” I mean the cost of getting up to speed and mastery of those tools.) There’s also an additional cost in the sense of the time it takes you to get those tests built, running, and maintained over time.

You have to be open and frank about the monetary costs of testing. It’s absolutely worth it, but get everything out in the open at the start so folks aren’t surprised later on.

You’ll need long-term, honest commitment

Testing requires commitment by everyone on the team at all levels. Every role from PM through dev and tester through stakeholder has to accept the many things I’ve laid out above—and they’ve got to commit to supporting the team. There will be times when management pushes back on time and resource commitments around testing. You and your team have to figure out when to fight that and when to realize that business has its own set of needs and that middle ground is an acceptable place to be.

Just remember the most important goal: Building great software that helps your customers do great things.

Development is part of that goal. Testing is too!