31 Days of Testing—Day 10: Mocking Out Dependencies

Updated: Index to all posts in this series is here!

Frankly, I’ve dreaded writing this post. Mocking is something that’s difficult to clearly get across, and the approaches and tooling for mocking vary perhaps even more widely than those for testing frameworks. There’s also an extraordinary range of terms people use to describe subtly different aspects of mocking—because we software folks don’t have enough confusing terminology in our lives already.

While I’m being specific with a mocking implementation in this post, it’s important that you keep focused at the higher level of the concepts I’m trying to get across. Look to the broad concepts and go do more exploration in the mocking tools and approaches specific to your platform.

Enough disclaimers. Let’s roll.

Why Mock?

In testing, mocking is a technique to let you cheat your system to isolate away dependencies external to the area of code you're trying to test. This is particularly important in unit testing where you want to focus on one small portion of your system—and you can’t do that if you’re having to deal with calls to external services, databases, security providers, etc.

Let’s have a look at one example. Below is some code for updating an employee’s information. This sort of action can be quite sensitive, especially since there is privacy and salary information involved. Ergo, you want to ensure there’s very solid security around this use case. My example below uses a security provider service which checks the current user and sees whether that user is permitted to do the specific actions on the target employee.

public class EmployeeUpdater

{

    private readonly IUpdateEmployeePermissible _updateEmployeeSecurityProvider;

    public EmployeeUpdater(

        IUpdateEmployeePermissible updateEmployeeSecurityProvider)

    {

        _updateEmployeeSecurityProvider = updateEmployeeSecurityProvider;

    }

    public Employee UpdateEmployeeRate(Employee targetEmployee, float newRate)

    {

        Employee updatedEmployee;

        if (_updateEmployeeSecurityProvider.CanUpdateEmployeeRate(targetEmployee.EmployeeId))

        {

            updatedEmployee = new Employee(targetEmployee.EmployeeId, targetEmployee.FirstName

                                           targetEmployee.LastName,

                                           targetEmployee.Address, newRate);

        }

        else

        {

            throw new SecurityException(

                "Invoking user doesn't have permissions to update target target employee. "

                + targetEmployee);

        }

        return updatedEmployee;

    }

}

I’m leaving out the implementation details for the security provider in an attempt to keep this as readable as possible. What’s important is an external (to this class) service figures out whether or not the current user can modify the target employee.

Design Impacts

Depending on your tool and platform, mocking may impact how you design your system. Note the security provider is passed in to the EmployeeUpdater’s constructor. Injecting dependencies this way, or through some form of setter, is an extremely preferred design approach for many reasons.

First, the EmployeeUpdater shouldn’t have to know how to create anything. Single Responsibility Principle says this class should know how to do its job really well, and not worry about other things. If the manner of creating a new security provider changed, then we'd have to modify the EmployeeUpdater, too. That’s nuts. Let things get built elsewhere and simply passed in to objects that need them.

Secondly, injecting the dependency to the class gives us a seam where we can substitute the real item with a fake one. We can use mocking tools, or just flat code, to create that faked out item and pass it to the portion of the system we’re trying to test.

One more consideration for mocking which can impact your design: different mocking toolsets on different platforms have limitations around what sorts of objects/classes they’re able to work with. In the .NET world, for example, many mocking tools are unable to deal with static or concrete classes. You’ll need to provide interfaces or abstract implementations for the classes you’ll want to mock out. Some tools will deal with static classes, some will deal with concretes, some won’t at all, and then some platforms lend themselves to completely different approaches.

Simple Stub Mocks

There are a number of different types of mocked objects. Mocked objects may have some behavior defined in them, or they may just be empty shells which do nothing more than satisfy your test’s need to have something in the “shape” of the real object.

In my examples below I’m using RhinoMocks for .NET. RhinoMocks uses the term “stub” to refer to a mock with no behavior, and “mock” for a faked object which has some behavior defined for it. The first test I’ll show you simply checks that an Employee object is returned with the correct updated pay rate. I don’t want to deal with the external security provider on this; the test is solely focused on whether or not the update happens properly. Ergo, I want to decouple the system and ignore the real security provider—I just want that to tell the system, “Yes, the current user is authorized.”

[Test]

public void Updated_employee_has_correct_rate()

{

    int userId = 1;

    int oldRate = 5;

    int newRate = 10;

    var stubProvider = MockRepository.GenerateStub<IUpdateEmployeePermissible>();

    stubProvider.Stub(x => x.CanUpdateEmployeeRate(userId)).Return(true);

    Employee current = new Employee(userId, "Jim", "Holmes", "Doghouse", oldRate);

    EmployeeUpdater updater = new EmployeeUpdater(stubProvider);

    Employee updated = updater.UpdateEmployeeRate(current, newRate);

    Assert.AreEqual(updated.HourlyRate, newRate);

}

The first statement in this test creates a stub of the security provider as defined by the interface IUpdateEMployeePermissible. (Dear .NET folks: can we please lose the last vestige of the silly Hungarian notation and just drop the “I” from interface names?).  The second statement defines an implementation that says “Whenever your method CanUpdateEmployeeRate is called, return true.”

Next we create an Employee object as test data and an EmployeeUpdater to actually test. Note that we pass in our stubbed out provider to the constructor. The EmployeeUpdater will now use the fake provider we created earlier.

The rest of the test is straight forward: invoke the method, check the results.

Because we’re using a stub, the fake provider won’t even care if we invoke CanUpdateEmployeeRate with a bogus ID. It will simply return true regardless of when or how it’s invoked.

Mocks With Behaviors

Now for the next test. We want to ensure that our security provider’s CanUpdateEmployeeRate is always invoked—this is a critical part of our overall business rules to ensure proper security for updating employees, after all.

What would happen if someone was working on the CanUpdateEmployeeRate method and commented out the security check for some troubleshooting, then mistakenly checked that in to source? You’d completely bypass the security check in production. Ick. Like that sort of thing never happens…

Mocking frameworks generally provide you ways to deal with this sort of interaction. In RhinoMocks you can use a “mock” to handle this.

[Test]

public void Verify_security_provider_is_called()

{

    int userId = 1;

    int oldRate = 5;

    int newRate = 10;

    var mockProvider = MockRepository.GenerateMock<IUpdateEmployeePermissible>();

    mockProvider.Expect(x => x.CanUpdateEmployeeRate(userId)).Return(true);

    Employee current = new Employee(userId, "Jim", "Holmes", "Doghouse", oldRate);

    EmployeeUpdater updater = new EmployeeUpdater(mockProvider);

    Employee updated = updater.UpdateEmployeeRate(current, newRate);

    mockProvider.VerifyAllExpectations();

}

Several major differences here: First, we’re generating a Mock, not a Stub in the first real statement. Secondly, we’re calling mockProvider.Expect instead of .Stub. The Expect call sets an expectation that CanUpdateEmployeeRate will be called once with the specific value set in userId. The next several lines are all the same; however, the final statement in this method isn’t the usual Assert we associate with tests – it’s a call to the mocked out security provider’s VerifyAllExpectations method.

This final statement asks the mocked object to check whether or not the system under test did the things we expected it to. In this case, we expected the system under test to invoke the CanUpdateEmployeeRate method with a value of 1.

The verification will fail if the CanUpdateEmployeeRate wasn’t invoked at all, or if it was invoked with a value other than 1.

Mocking frameworks can provide a wide range of expectation checks. You can look to ordered expectations (method A called before method D), constraints (input value between x and z), and many other validations.

Tools for Creating Mocks

You might start to see that creating a series of mocks could be quite complex. For example, if my security provider depended on something else which in turn depended on something else, then I’d have a nasty tree of objects I needed to deal with. Tools exist in every platform to help ease this. Factory Girl in Ruby is one, StructureMap and Ninject in the .NET world are others.

Look up these approaches to help you ease the burden of dealing with complex object graphs both in your testing and production worlds.

Wrapping Up

This is an extremely simplistic, high-level view of mocking. My goal was to simply show where mocks might be used, and expose you to a few of the fundamental concepts around them. Mocks are an extremely powerful tool, but it’s extremely easy to use them too much, or in too confusing a manner. It’s also easy to end up testing the mock you just created instead of the system itself!

Don’t stop with this post. Go read more around mocks and figure out how they can help you better test your systems!

31 Days of Testing—Day 9: Readable Tests

Updated: Index to all posts in this series is here!

This post is an off-the-cuff tangential post based on some interesting comments to my previous posts on test fundamentals.

A couple folks commented on my use of C#’s “var” keyword, pointing out that it made the tests less readable in their view. I think it raises an interesting point of discussion that is an extremely large debate on the value of strict typing. I’m going to forgo that conversation, and instead try to focus down on readability of tests.

In the following test, what’s most important?

[Test]

public void Computing_with_40_hours_at_rate_5_returns_200()

{

    WageComputer computer;

    bool isHourlyWorker = true;

    var computedWages = computer.ComputeWages(40, 5, isHourlyWorker);

    Assert.AreEqual(200, computedWages);

}

This test is validating that the ComputeWages method is correctly working and returning the correct values for my inputs. This test isn’t looking to validate the return type. Frankly, I don’t care what the return type is. I’m more concerned that the algorithm is functioning properly.

If I’m only concerned about the value coming out of the method, then, quite frankly, anything extra at all on the left-hand side of that statement is pure noise to me. Look at these two different examples:

double computedWages = computer.ComputeWages(40, 5, isHourlyWorker);

var computedWages = computer.ComputeWages(40, 5, isHourlyWorker);

That’s how my mind looks at these sorts of things. Frankly, I’d be happier if the compiler just got rid of the entire type declaration. The compiler knows what the right-hand side is returning. Even “var” is just noise to me. Something like this would be a thing of beauty:

computedWages = computer.ComputeWages(40, 5, isHourlyWorker);

(By the way, that’s not my idea. My pal Leon Gersing put this idea in front of me during an interesting discussion several years back at a .NET training day we put on.)

Moreover, explicitly using strong typing makes your tests more brittle. If you ever change the return type from ComputeWages you’ll need to go back and update every test using that class. (In this case that’s a somewhat contrived example. Please look at the larger point here.)

If you feel the need to have a test specifically checking the return type, then write a separate test for that.

Carry this over to everything else about your tests. What’s noise? What’s really important? Can you easily grok a test you wrote three weeks ago? Three months ago?

No? Why not?

Great software engineers, or the latest trend of labeling folks who care as “craftsmen,” understand the criticality of writing code that’s easily readable.  Tests need to have the same care, because tests are production code.

Take a close look at what you really for your test and cut out everything else. It’s just noise.

Updated: By the way, in case you get put off my my frequently curmudgeonly sounding tone, I really do love this sort of feedback and questioning on my posts. I do not have all the answers. Your questions and points make me constantly re-evaluate how I’m doing things, and that’s a good thing! Please, keep up comments and questions. I love ‘em!

31 Days of Testing—Day 8: Pay Attention to Your Tests’ Setup!

Updated: Index to all posts in this series is here!

In yesterday’s post I laid out some fundamentals of what a test looks like in C# with NUnit, plus how you’d go about running it and getting results. Keep in mind that while I’m using NUnit and C# to illustrate fundamentals for these few columns, the basic tenets apply regardless of your platform or testing framework/toolset.

Today I want to focus on handling setup conditions for your tests. In yesterday’s examples, every test created a new WageComputer instance. This sort of duplication gets tedious and can lead to more maintenance hassles since it violates the DRY principle (Don’t Repeat Yourself). If you scatter creation of prerequisite classes, services, data, etc. across multiple tests, you’ll have to go to those same multiple tests to fix busted instantiations when those prerequisites change.

Instead, look to push this sort of work off to a centralized location. In many cases, you can use your test framework’s features to do this. NUnit and many other frameworks support setup methods at the fixture and namespace scope. Moreover, you can have these setup methods run once per class/fixture, or once per test. This lets you have great control over where you create prerequisite objects, load/create data, etc.

Here’s how this looks in NUnit using my examples from the previous post:

[TestFixture]

public class When_working_with_an_hourly_worker

{

    private WageComputer computer;

    bool isHourlyWorker;

    [TestFixtureSetUp]

    public void Run_once_before_any_tests_run()

    {

        isHourlyWorker = true;

        computer = new WageComputer();

    }

    [Test]

    public void Computing_with_40_hours_at_rate_5_returns_200()

    {

        //Arrange

        //Act

        var computedWages = computer.ComputeWages(40, 5, isHourlyWorker);

        //Assert

        Assert.AreEqual(200, computedWages);

    }

    [Test]

    public void Computing_with_41_hours_at_rate_5_returns_207_5()

    {

    //Arrange

    //Act 

    var wages = computer.ComputeWages(41, 5, true);

    Assert.AreEqual(207.50, wages);

    }

The TestFixtureSetup attribute will cause the NUnit runner to execute the method Run_once_before_any_tests_run when this class/fixture is first loaded. We’ll stand up a new instance of the WageComputer and set isHourlyWorker true. This is very similar to a class constructor, but the lifecycle is managed by the test framework, not the .NET internals. (Which I couldn’t explain to you. Go ask Jon Skeet or Bill Wagner.)

There’s a similar attribute called SetUp which executes before each test. This is handy if you need to freshly initialize something before each test. (Remember, tests shouldn’t rely on any state set in another test.)

What gets set up may need to get torn down. Frameworks and tools generally support some form of TestFixtureTeardown or Teardown approach to clean up after each fixture or each test, respectively. These are great places to stuff transaction rollbacks to clean up after database interactions, for example.

This is a very simplistic, trivial example, but I’m sure you grok the general concept here. You can use fixture setup and teardown methods to deal with fixture-related prerequisites. Very handy.

Now for some important caveats.

While you want to avoid too much duplication, you can easily get carried away and overly clever with inheritance and abstraction of your setup actions—to the point where it gets extraordinarily difficult to understand what’s being set up where.

In a previous life I worked with an internally created test framework based off a Behavior Driven Development framework. We let ourselves get perhaps a little overly clever with our inheritance and setup chain, and it became very difficult to learn and understand. (OK, there’s no “perhaps” about it. We did, and I was a major part of letting that happen. Bad Jim. Bad Jim!)

Using some psuedo code, the inheritance and setup chain looked something roughly like this:

Integration_test  (Base)

  //sets up database

Functional_test (Inherits from Integration_test)

  //sets up browser, configures UI

Feature_test  (Inherits from Functional_test)

  //configures system

Module_test  (Inherits from Feature_test)

  //creates a module

//Now we get to the actual tests!

When_creating_something_as_regular_user  (Inherits from Module_test)

  //creates new user

  //FINALLY does some testing!

Imagine you’ve got a failure in a test. You could potentially have to walk back up four layers of inheritance to understand exactly what’s going on where. This is an overly clever approach to setup which makes understanding of your tests extremely difficult. Remember that code is read and re-read many more times than it’s written.

In these sorts of cases it’s OK to relax a bit on the Don’t Repeat Yourself principle. There’s an extremely applicable saying for this situation: “Keep your system DRY and your tests slightly moist.” What that means is, it’s OK to duplicate some setup steps in your tests if it makes the fixture/class/spec more understandable.

The Bottom Line

Leverage your testing platform/framework’s features for helping you get prerequisites for your tests set up, just don’t get so convoluted that you can’t easily understand what’s going on in the test when you open it up a couple weeks or months after you’ve written it.

31 Days of Testing—Day 7: Automated Test Basics

Updated: Index to all posts in this series is here!

I thought today’s post would be a good opportunity to lay out some fundamentals of automated tests work. The mechanics of test automation varies extraordinarily based on what platform you’re working on, and even within platforms you’ll see vastly differing approaches based on whatever testing framework/toolset you’re using.

Testing frameworks have a number of commonalities, regardless of platform and implementation. You’ll have a test runner which executes the tests. Test runners may live in the IDE you’re using to write your tests, they may be a separate GUI, or they may be a command line runner. Command line test runners make it possible to run your tests in a number of different environments such as a build or continuous integration server. The tests themselves will likely be organized, depending on the framework/tool, into classes, fixtures, scenarios, or something else. You generally point your test runner at these fixtures and the runner noodles out what tests are contained within those groups. The runner will execute all those tests and give you back a report on the pass/fail status.

Today’s column shows unit tests, but the general concepts apply to integration and functional tests too.

I’m going to be showing examples in C# with NUnit and RhinoMock, simply because that’s what I’m very comfy with. Yes, yes, we should all get out of our comfort zone and do things on other platforms, but the point of this series isn’t for you to watch me flail around while trying to learn new stuff…

Some things to keep in mind as you read posts of mine which contain code:

• I’m old school, although I try hard to keep up with new trends
• I haven’t written system code in a long time, just test code. There’s a bit of a difference.
• I am a starting point for your learning, not a destination. Look at my stuff, figure out what works and doesn’t, then go find other places to learn more.

The examples I’ll be using today are taken from the Unit Testing 101 talks I give. The examples aren’t *DD-ish because I want to focus on very fundamental pieces with out adding any methodology to the mix. Like I said, old school—but the foundational principles are critical and similar regardless.

Without further ado, part one, unit test basics. The system we’ll be testing is a simple payroll wage calculator method. It’s purposely not optimized or concisely written because I use it as a starting point for some refactoring, etc. (If you’ve interviewed with me for a job you’ve likely seen a variant of this…)

Here’s the method:

public float ComputeWages(float hours, 

                            float rate, 

                            bool isHourlyWorker)

{

    if (hours < 0)

    {

        throw new ArgumentException("Hours must be greater or equal to zero " 

            + hours);

    }

    float wages = 0;

    if (hours > 40)

    {

        var overTimeHours = hours - 40;

        if (isHourlyWorker)

        {

            wages += (overTimeHours*1.5f)*rate;

        }

        else

        {

            wages += overTimeHours*rate;

        }

        hours -= overTimeHours;

    }

    wages += hours*rate;

    return wages;

}

This method figures wages for hourly and salaried workers based on their status (hourly/salaried), number of hours worked, and their hourly rate. Salaried workers don’t get overtime; hourly get time-and-a-half for anything over 40 hours.  We also check for negative inputs around hours. There’ input guards we should handle, but again, this method purposely leaves off a number of things.

Let’s cover basic tests for an hourly worker first. Looking at the code we want to make sure we hit all the boundaries in this method. There’s a boundary for over/under 40 hours, so we’ll need values over and under that. I always like to specifically test zero as well, so we’ll want a value there. Finally, we’ve got a check for negative hours, so we’ll need that. Let’s just use five bucks per hour as a our rate to simplify things. Here are our input values and expected outputs.

Hours	Rate	Expected
0	5	0
40	5	200
41	5	207.50
-1	5	ArgumentException

Let’s start with the simple happy path test first.

I mentioned fixtures above as a container for individual tests. In NUnit you create a class and decorate it with the TestFixture attribute. Other frameworks work in a similar fashion. Mostly.

[TestFixture]

public class When_working_with_an_hourly_worker

{...

A test fixture should contain a logical group of individual tests. Styles of grouping and naming conventions vary dramatically between advocates of particular methodologies. I’m avoiding all that discussion here and just focusing on the fact a fixture holds tests. You need to figure out what works with your team in your environment on your project.

In NUnit a test is simply a method with public scope that’s decorated with a Test attribute:

[Test]

public void Computing_with_40_hours_at_rate_5_returns_200()

{...

Now it’s time to write the actual test. Patterns for writing tests vary with methodologies, but I think there’s a common pattern regardless: Arrange, Act, and Assert. You Arrange the things you need to run the test, you Act on the system you’re testing, and you Assert to see if your actual results are what you expected.

Asserts, regardless of how they’re named in your framework/tool, are simply equality checkers of one form or another. You’re checking two strings are the same, two integers have the same value, or two collections have the same values in the same order. You’re generally asserting these conditions are equal, although you’ve likely many options such as Assert.IsFalse(someBoolean), Assert.IsNotNull(somethingElse), etc.

Here’s what our actual test will look like:

[Test]

public void Computing_with_40_hours_at_rate_5_returns_200()

{

    //Arrange

    WageComputer computer = new WageComputer();

    bool isHourlyWorker = true;

    //Act

    var computeWages = computer.ComputeWages(40, 5, isHourlyWorker);

    //Assert

    Assert.AreEqual(200, computedWages);

}

The final statement in this test is the Assert: it’s comparing the expected value (200) to the actual value returned from the call to ComputeWages. You’ll continually hear the terms “expected” and “actual” in testing parlance.

I’ll use the JustCode test runner that lives within Visual Studio to run this test. Here’s what it looks like as a passing test:

If I change the expected value in the test to an incorrect value, we’ll see what the failure looks like. Note the red colors and the detailed message on the right frame. Your test runner’s output likely varies, but the point is you get a clear failure and an exact reason why the test failed. In this case, the expected value was 201 (my manual hack) and the actual was 200.

Really, the first three test cases are variants of this: input values and an expected output value. Let’s jump to the last of the test cases, where our expected result is an ArgumentException. Every testing framework should give you the ability to assert exceptions are happening as you expect. In NUnit that’s handled simply via an attribute. Here’s how that actual test looks:

[Test]

[ExpectedException(typeof(ArgumentException))]

public void Invoking_with_negative_hours_throws_ArgException ()

{

    WageComputer computer = new WageComputer();

    var wages = computer.ComputeWages(-5, 5, true);

}

You’ll note we’ve got some duplication in our two tests: we’re repeating the setup of our WageComputer class. This isn’t a huge code smell for two tests, but I’d rather not see us continue this needless duplication as we grow our tests out.

Tomorrow we’ll discuss how to deal with this situation by using setup methods, and we’ll cover their corresponding teardown equivalents. We’ll also dive in to data driven tests.

31 Days of Testing—Day 6: Types of Automated Testing

Updated: Index to all posts in this series is here!

Automated tests come in several flavors, and I sometimes hear folks mixing up terms for the type of test they’re discussing. I thought I’d take one short column to lay out a few definitions.

Unit Tests

Unit tests focus on one small piece of the system’s functionality, most often a method or procedure. Unit tests never cross any service boundaries, and shouldn’t even rely on other dependencies outside of the specific area being tested. Unit tests work in isolation—any external dependencies need to be faked/mocked/stubbed out. If your test is hitting the file system, database, or a web service, then it’s not a unit test.

Imagine a payroll computation method which takes hours worked, a worker’s hourly rate, and whether or not that worker is salaried or hourly. The method is a simple algorithm and doesn’t have any outside dependencies. A unit test would be one that calls that method with a set of input parameters (42 hours, $20 per hour, salaried worker) and checks the output of that against expected results.

Keeping true to the above-described isolation requires a well-designed system that enables you to mock out these dependencies. There are a number of design approaches that help this, but those things are well beyond the scope of this series. (Go look up dependency injection, inversion of control. Also check out tight coupling and use of static classes.) The act of mocking out these dependencies means you’ll be looking to tooling to help you out. Many frameworks exist to help you get through this on whatever platform you’re working on.

Because of this isolation, unit tests are blisteringly fast. It’s common for thousands of unit tests to run in a few tens of seconds. Unit tests are your first line of defense against regressions. Developers should be running them constantly through their work cycle, and most definitely before committing any work back to the source control repository.

Integration Tests

Integration tests specifically target interaction between different components or services. Integration tests are most often below the user interface level, checking things like business logic, persistence, or web services.

An integration test might call a web service to create a user, then call the database directly to insure the user was indeed created.

Integration tests are generally quite slower than unit tests because of the overhead involved with standing up web services, hitting databases, etc. Because of this slower execution speed integration tests aren’t generally run as often as unit tests. If your integration test suite is small enough you may be able to have your developers running them right before they commit their final work on the current work item. It’s more common to run only the set of integration tests directly relating to the work item, then run the entire suite of integration tests during regular smoke checks throughout the day. (Or once per night if your suite is really large and you’ve not set up your infrastructure for parallel execution.)

Functional Tests

Functional tests are the slowest of all the tests because functional tests are generally standing up the real application, either on the desktop or in a browser, and driving that user interface around to perform the same actions a user would.

An example of this would be using something like Selenium or Test Studio to start up a browser, open your web application, and take some actions to place an order in an electronic purchasing system.

A great many tools and frameworks can be used to help you write functional tests depending on what platform you’re working on. There are many flavors of functional tests, but at their core they’re responsible for helping you verify one specific test case around your system’s functionality behaves as expected.

Performance Tests

Performance testing is a broad umbrella encompassing a number of different types of tests. Performance, load, stress, failover, soak, endurance, and other terms all come in to play. I wrote a somewhat longish post on my blog at Telerik around these different types. You can go read that for more detail, but here’s the Cliffs Notes version:

• Performance: Measures the performance of one specific functionality slice, such as saving a user to the database. May be from the user interface down, or may be some slice of internal functionality.
• Load: Checks the system’s performance while ramping up the load on the system by hitting it with simultaneous connections from some source. Measures performance of a scenario under real-world loading.
• Stress: Can also be called failover testing. Can also be called “Throw an insane amount of traffic at the system, then throw more traffic at it until something breaks.” Less flippantly, the idea is to ramp up traffic and load on your system until it degrades or outright fails. The idea is to understand what the maximum useable load is, and what happens when that’s exceeded.
• Endurance: Systems behave differently under load over a long period. Things like subtle memory leaks show up which you might not otherwise find.

Security Tests

Automated security tests are looking for regressions in protections around cross site scripting, SQL injection, and any of the other numerous (and really scary) security-related threats.

Tools for automating these sorts of checks are quite specialized, and I don’t pretend to have a great grasp on them. Instead of possibly leading you astray, I’ll just recommend you look elsewhere for this information.

Each Type of Test Has Its Place

You could make the case there are other types of automated testing, but I think these categories above cover the most important types. What’s critical is that you understand the basic fundamentals of each type and its role.

(Note that I didn’t discuss testing methodology such as test-driven development, behavior-driven development, acceptance test driven development, etc. Those are methodologies which use the types of tests I described above.)

I’m a firm believer that you absolutely need all of these types of tests working together. I’ve seen far too many instances where a developer or team focused on getting unit tests rolling, only to find bugs that would have been caught at the integration or functional level. Performance testing may not be required on every project, but I’d make the case it’s valuable to have at least a bit of metrics around performance simply to ensure you’re not losing ground on user experience as you continue to evolve the system.

Know your test types. Use them in the right places.

31 Days of Testing – Day 5: Choosing What To Test

Updated: Index to all posts in this series is here!

You can’t test everything. Get over it.

Even if you’re doing some form of test-first development, you are not going to go as far down the testing rabbit hole as you might like. In my post Focus on the Why, Not the How  I passed on a true story of a pal who, while paired with an industry “expert,” spent five hours ripping apart a view to get 100% test coverage. The exercise turned out to be useless, and they delivered no value whatsoever that day because the remaining three hours of the day were spent rolling back the work they’d done to that point. #FAIL.

What you can do is carefully chose what to test in order to get the most effective testing possible. There are a great many ways to start narrowing down what you should look to hammer away with on testing. Most of this article is directed at exploratory or functional testing, but the points apply generally to all forms of testing--I hope!

In his chapter from Beautiful Testing Adam Goucher speaks of evolving to a focus of “uncover[ing] quality-related information as efficiently as possible.” (Emphasis Adam’s.) This is really a great line, because it’s exactly how I view testing. I want to get the best information about the state of the system, and do it as quickly as possible.

Adam has a great acronym he uses to guide his testing: SLIME, which stands for Security, Languages, RequIrements, Measurement, Existing. (He also freely admits he had to stretch the heck out of “requirements” to make it fit his wonderful acronym.) This is one great way to approach what and how to test existing systems, or to do exploratory or additional testing on a system being developed.

I look at a number of similar things in determining what I want to have a look at, or where I want to focus my automation efforts. (Again, this is in addition to tests that are hopefully being written by devs and testers pairing together as the system is being built!)

Choosing by Value

Hopefully your team is focusing very tightly on delivering only valuable features to your customer. Look at those features, or the existing functionality, and prioritize the most critical things first. You’ve likely got a mix of features and work items driven by the market and customer. Prioritize that list, and figure out which are the most critical to spend your time on. Within that list, figure out how far down the rabbit hole it makes sense to dive.

Dealing With Regulatory Compliance

Perhaps you’re under constraints mandated by some form of legal or regulatory compliance. In that case, you’ve got to do some disciplined focus to ensure you’re meeting those legal constraints. HIPAA, Sorbanes-Oxley, privacy-related information—all these domains will absolutely require your focus if you’re working in those environments.

You may be forced to drop all other work and focus on the features under that constraint to ensure you don’t put your entire organization at risk. Tom and Mary Poppendieck speak to just this situation in their book Lean Software Development—they had a state-mandated set of regulations on a project they were brought in to rescue. Everything else had to be put on the back burner in order to ensure they met the legal requirements.

Choosing by Risk

Another approach is to look to your riskiest features/work items/components and focus on those first. Are you working with a complex security model in your system? Do you have an extremely large, complex feature your team’s been under pressure to roll out? What about overall complexity of the component or feature?

Focusing on your highest risk areas first will help you mitigate last-minute panic when that complex system doesn’t work as expected – and you’re only a couple days away from release. Focusing on those risky areas gets those problems identified as early as possible.

Choosing by Measurements

Looking to concrete measurements is a great way to figure out where to focus additional testing effort. You don’t want to spend time on areas that are already covered by tests, or which have been stable in the past, so look to code coverage tools and bug reports to help you find areas to possibly cover.

Code and architectural/design complexity are great places to look at for determining where to add more testing. Most every platform has a set of tools which will give you stats on things like cyclomatic complexity, lines of code, coupling, etc. These stats are not only geeky cool, they can point you right at spots rife for bugs.

Finally, have a look at code churn, the amount of changes made to classes, files, or components in your system. Lots of churn means a lot of work has gone in to that item, and the chances are something may be amiss. Churn coupled with code metrics and bug histories makes for an awesome indicator!

Choosing by Spidey Sense

Finally, there’s your intuition. You know what your team is like. You likely have a feeling for areas of the system that have scary corners. You know which areas of the system have given you grief in the past.

Trust your spidey sense and hit those areas for some rough and tumble exploratory testing. Use those bugs to create new integration/functional/unit tests around what you discover and fix.

You can’t test everything. You’ve got to apply some discipline and planning so that you can do your testing as effectively and efficiently as possible.

NOTE: I encourage you to go read up on works by folks like Lanette Creamer, Elisabeth Hendrickson, and others who are doing amazing things in the exploratory/rapid testing domain. There are a lot of tremendously smart folks out there doing great work!

31 Days of Testing - Day 4: Sustainable Pace, Sensible Flow

Updated: Index to all posts in this series is here!

Updated: Fixed some busted paragraph formatting. Sorry for the massive run-on post!

If you want to continually deliver great software in a rapid fashion at as low a cost as possible, then you need to step back and understand how critical sustainable pace and sensible flow are—especially as it relates to testing.

Sustainable pace is a critical rule that needs to be part of a team’s foundation. You work a sane amount of hours every day, then go home and come back the next day refreshed and ready to do great work. Yes, you may need to work occasional spurts of extended hours, but that needs to be the very, very rare exception. Environments without a fundamental belief in sustainable pace keep silly amounts of pressure on their teams to complete work in unreasonable periods. “No, we’re not cutting scope. We’re in this mess because you created too many bugs previously, and our marketing department has made too many promises. Work harder. Failure to meet the release date with the promised scope is not an option.” Such attitudes and environments will kill your team over time, and will end up driving the quality of the products/systems into the ground over time. Your developers speed through their work and throw everything over the fence to the separate QA group. The separate QA group is overwhelmed and ends up feeling like scapegoats for the release’s poor quality.

Sensible flow is a term I’ve been using, off and on, to describe how work items need to move through your development process and environment. I’m sure other smart folks in the industry have other terms, but it’s what I’m using to describe things like ensuring you have great collaboration, proper work item sizing, and honest measurements or standards for what “done” really means.

The notion that testing is a separate activity leads to testing being the last bit of work done in iterations/sprints/release cycles. This is madness, quite frankly. You’re putting an immense amount of unreasonable pressure on your testers, and it also creates additional stress when they inevitably reject work items with only a day or two to do the rework and revalidation.

I’ve worked in these environments in the past, and I continue to talk with a lot of folks in the community and clients who are in these environments now. My biggest, bestest suggestion to them: Take the mental step of throwing away the notion that testing is separate from development.

That may seem like a trivial thing to do, but it really helps you, your team, and your organization to start being honest about when a work item is really done. That item isn’t done until it’s been tested. (I could go a few steps farther and say it’s not really done until it’s released to production, but as with all these columns I’m trying to save you from too many of my wandering rants.)
Agreeing that an item has to be tested before it’s considered done means you have to be honest about including schedule time for testing. This discussion is a great positive factor when you’re deciding what you can accomplish during your iterations because your team will be more consistent about identifying possible bandwidth problems near the end of the iteration.

Once you get honest about having to include testing as a measure of an item being done, then you have to get honest about how you’re going to meet done for your work items in your iteration. Now we can start to look at how you’re sizing your work items.

If your work items are too big, then they get hung up in the development stage for too long. Testers don’t get their hands on the work items until late in the cycle, and there’s little time left for finishing up automation and doing exploratory testing. (Note I said “finishing up automation” because I believe your devs and testers should be pairing up on building automation as part of the development cycle!) These delays lead to pressure to get everything done in the few days at the end of the iteration. Folks work longer hours and/or work doesn’t get finished up within the iteration. Both are bad.

You need to figure out what size work items works well in your environment, but I’d encourage you to try to reach work items that can be completed in a day or two. Yes, by “completed” I mean done. That’s designed, discussed, developed, and tested.

Small-sized work items mean you’re able to start getting a great flow through your team’s process. There’s much more sense of progress, and there’s less impact when a work item fails the testing stage and has to be put back in for further rework. The team sees a lot more accomplishments, even if those items are at a smaller scale.

Now you’re open and clear about what done means, and now you’re working hard to properly size your work items. Ensuring sustainable pace and sensible flow means watching your processes stages to avoid pileups and bottlenecks. Maybe your development team is crushing their work and getting through the development portion of work items at a tremendous pace. The downside of this is you’re stacking up a huge amount of Ready-To-Test work items in the testers’ area. What do you do?

Look at this backlog pileup as a team problem. Solve it as a team problem. Have your developers roll up their sleeves and help work through those items. No, developers aren’t testers, but they’re quite smart and can do a great job helping work through this backlog. A couple ground rules do have to be in place for this to succeed, though. First, no dev can test a work item they did. Secondly, each dev has to meet with a tester for a few moments before they test to quickly discuss testing strategy for that work item. Lastly, the dev has to quickly review their actions and discoveries with the testers after they’ve done their work.
Once you get back to a good spot with your pipeline’s backlog then developers can go back to their primary tasks.

Focusing on keeping a sustainable pace, and a sensible flow ensures the software you’re building gets solid testing throughout the process. Be honest about what done means, size your work properly, and work as a team to avoid backlog pileups.