31 Days of Testing—Day 19: Refactoring a “Monster” Functional Test, Part 1

Note: A day late on this post. My daughter had her tonsils removed yesterday, so I was rather caught up with life.

Index to all posts in this series is here!

Folks new to functional test automation can often times dive straight off the deep end when trying to get started. I’ve often seen examples of “my first test!” where those test scripts/files/fixtures are hundreds of steps long and test many different cases. They’ll start out with one script that starts the browser, loads up the site, logs on as an administrative user, creates some users and test data, logs on in a different role, takes some actions using the setup data, does some validations, then logs back on as the administrative user and cleans up the data created to run the test.

A good automation test fixture/case/script is concise and focused on testing one specific thing. Data setup should be handled in either helper scripts or pushed off to a backing framework that calls stored procedures, web services, or internal APIs to perform setup data. Good cases shouldn’t generally be switching between roles; that same sort of configuration or prerequisite actions should be handled again by helpers or APIs.

These “monster” automation scripts suffer from exactly the same maintainability and functionality issues that bad code in the system’s codebase does: high complexity, mixed concerns, duplication of functionality, etc. Monster scripts are nothing but a recipe for pain—regardless of the good intentions by the folks who created them in the first place.

Over the next few blog posts I’m going to take an example of one of these monster scripts and break it apart into a better set of more focused test cases and support modules. I’ve had to do this more than once with scripts written by customers, contacts, pals, and me.

Aufpassen! Before you dive in to this kind of refactoring, step back and see if it’s a worthwhile use of your time. You may likely be better off just rewriting the dang thing. I’m rolling with it now because it’s an interesting exercise and I hope you’ll find it useful.

For this set of posts I’ll be using the demo app my group at Telerik had built to help us demonstrate specific testing problem areas like AJAX, multiple windows, etc. You can find the demo app hosted out at Heroku. Logon credentials are testuser/abc123 if you feel like playing around a bit.

This example is using Test Studio, but the concepts are absolutely the same regardless of what testing framework/tool you’re using.

Let’s start off by discussing what the intended test is: validating that a newly created user can successfully be retrieved from the system. We might express that test case in the form

• Given I have created a new user with input data of <a list here>
• When I click on that user in the main grid
• Then I should see the new user displayed with their correct data.

I built a single script to run this test soup to nuts – exactly the sort of “monster” script I talked about above. In Test Studio this iteration of the test has 32 steps (33 lines below because of a continuation). I did a little transform-fu of the test file to come up with some :

   1: Navigate to : 'http://localhost:3000/'

   2: Click 'LoginLinkLink'

   3: Set 'UsernameText' text to 'testuser'

   4: Set 'PasswordPassword' text to 'abc123'

   5: Click 'LoginButtonSubmit'

   6: Click 'NewContactLink'

   7: Connect to pop-up window : 'http://localhost:3000/contacts/new'

   8: Set 'ContactFirstNameText' text to 'New'

   9: Set 'ContactLastNameText' text to 'User'

  10: Set 'ContactEmailEmail' text to 'new.user@foo.com'

  11: Set 'ContactLinkedinProfileText' text to 'http://linkedin.com/newuser'

  12: Check 'ContactGovtContractCheckBox' to be 'True'

  13: Check 'ContactDodCheckBox' to be 'True'

  14: Check 'ContactOtherCheckBox' to be 'True'

  15: Desktop command: Drag & Drop Neutral Lead Image to Lead Type Drop Target

  16: Click 'CommitSubmit'

  17: Wait for 'TextContent' 'Contains' 'New' on 'NewTableCell'

  18: Verify 'TextContent' 'Contains' 'User' on 'UserTableCell'

  19: Verify 'TextContent' 'Contains' 'new.user@foo.com' on 'NewUserFooTableCell'

  20: Verify 'TextContent' 'Contains' 'http://linkedin.com/newuser' on 'HttpLinkedinTableCell'

  21: Verify attribute 'alt' has 'Contains' value of 'Neutral' on 'New User Lead Type'

  22: Extract attribute 'href' on 'ViewContactLink' into DataBindVariable $(ContactLinkUrl)

  23: Coded Step: [Retrieve_a_newly_created_user_and_validate_users_data_is_correct_CodedStep1]

  24: Click 'ViewContactLink'

  25: Coded Step: [Retrieve_a_newly_created_user_and_validate_users_data_is_correct_CodedStep]

  26:             Connect to pop-up window : 'http://localhost:3000/contacts/7', ConnectToPopup=True

  27: Verify input 'ContactFirstNameText' value 'Exact' 'New'.

  28: Verify input 'ContactLastNameText' value 'Exact' 'User'.

  29: Verify attribute 'value' has 'Same' value of 'new.user@foo.com' on 'ContactEmailEmail'

  30: Verify input 'ContactLinkedinProfileText' value 'Exact' 'http://linkedin.com/newuser'.

  31: Verify attribute 'alt' has 'Same' value of 'Neutral' on 'LeadTypeImage'

  32: Close pop-up window : 'http://localhost:3000/contacts'

The test logs on (lines 1-5), creates a new user (6-16), verifies the user was correctly created on the main grid (17-21), figures out which link on the main grid to click to open the new user in the edit contact form (22-23), opens the new user in the edit contact form (24-25), then validates all the new user’s info is as expected (26-31). Finally we close out any new browser windows that were spawned.

Things would look very similar if I was using Watir, Selenium, or some other testing tool. The syntax would obviously be different, but the idea of stringing together huge numbers of steps in a functional test is a common one, and no framework or tool automatically fixes this issue for you…

Here’s a video of the test in action:

Before diving in to clean this mess up, let’s take a step back and figure out some goals to get us back in line with how a good script should work:

• Setup / configuration / prerequisites done outside the browser in helpers or internal APIs
• No duplication, or at least minimal duplication carefully selected for clarity
• Minimize navigation or unneeded steps
• Careful bullet-proofing of the tests

Before I do anything else, I’ll ask myself: is this test worth working on? Answer, yes, firstly because if not, then I’d have to figure out something else to write this blog post on. Secondly, this test has plenty of things of value in it which can get moved out to other useful steps.

After a bit of planning, here’s how I’m going to run with this:

1. Move the steps for logging on to a separate test (define the action once, reuse it as needed)
2. Move the steps for creating a new user out to an external API call

Let’s focus on the first thing: modularizing the login action. It’s easy to separate out that action. In a code-based framework I’d move that off to a new Page Object. In Test Studio I’m moving it off to its own test.

   1: Navigate to : 'http://localhost:3000/'   

   2: Click 'LoginLinkLink'   

   3: Set 'UsernameText' text to 'testuser'   

   4: Set 'PasswordPassword' text to 'abc123'   

   5: Click 'LoginButtonSubmit'   

   6: Verify 'TextContent' 'Contains' 'Logout' on 'LogoutLink'

   7: Verify attribute 'href' has 'Same' value of '/logout' on 'LogoutLink'

One additional step here. Generally I believe in avoiding conditionals in tests—that usually indicates you’re looking at two different tests. However, long experience in making a logon function modular has made me a believer in first checking that you actually need to log on—you may already be logged on, and the test would fail, breaking your script.

Do this by first checking whether the login link appears. If that’s there, log on. Otherwise, take no action.

   1: IF (Verify Exists 'LoginLinkLink') THEN

   2:     Navigate to : 'http://localhost:3000/'   

   3:     Click 'LoginLinkLink'   

   4:     Set 'UsernameText' text to 'testuser'   

   5:     Set 'PasswordPassword' text to 'abc123'   

   6:     Click 'LoginButtonSubmit'   

   7:     Verify 'TextContent' 'Contains' 'Logout' on 'LogoutLink'

   8:     Verify attribute 'href' has 'Same' value of '/logout' on 'LogoutLink'

   9: ELSE

In the next post I’ll work on moving out the user creation to an external API call!

31 Days of Testing—Day 18: Baseline Datasets

Index to all posts in this series is here!

Yesterday’s post by Seth Petry-Johnson on Data-Driven Tests got me pondering another aspect of test data: baseline datasets. I love baseline datasets, but you need to carefully think when and how to use them.

Why Use Baseline Sets?

Seth’s spot on in his assertion that tests should be responsible for setting up their own test data. Most of the time.

In some cases, however, it makes terrific sense to have a pre-built set of data carefully constructed to cover specific scenarios. Here are a few scenarios I’ve run in to in the past where a baseline dataset has really helped us out:

• Manual testing. Baseline datasets can be a great help in a couple areas of manual testing.
  • Setup. Yeah, I really want to spend three days setting up data by hand so I can run through effective exploratory testing, or through our manual testing guides. NOT! A baseline dataset gets me rolling right away.
  • Visual UI validation. A baseline dataset with large sets of data is also a great help for validating your UI handles lots of data well. Paging of grids? Long threads of conversations? Thousands of users in your site’s admin section? Look to your pre-built data to help here!
• Load testing. You can’t be serious about your load testing if you’re doing it against an empty or nearly empty database. You’ve got to have a realistic set of data in order to understand how your data access layer/module/strategery works, and how your business and UX layers deal with things.
• Automated testing. Yes, even your regular tests can be helped out greatly with a baseline dataset. Sure, your tests need to handle their prerequisites, but you can also look to have supporting data pre-configured—users in place, content created, etc.
• Testing BI, reporting, or data analysis systems. Systems that process large amounts of data need, well, large amounts of data to test with. It’s insane to think your test harness/suite/fixtures can create this sort of infrastructure each time you need to run your tests. Look to a baseline dataset!
• Ease of test setup and teardown. Yes, tests should clean up after themselves, but this can be extremely hard to do and quite brittle. You know what the easiest way for your test suite to handle that clean up is? DROP DATABASE is a thing of beauty in many scenarios.

Considerations for Baseline Datasets

Be careful, very, very careful, when moving to implementing a baseline dataset. While baseline datasets are extremely useful, you need to make sure you construct them to meet specific scenarios, and you need to make sure you understand how these datasets will impact all aspects of your testing.

Here are some questions I’ve asked myself and my teams when we’ve dealt with standing up baseline datasets in the past.

• What scenarios are we testing for? Make sure you know the test cases you’re trying to cover. Don’t just go creating scads of content in your database. Create sets of data to solve specific test needs, otherwise you’ll potentially risk side effects. (See below.)
• How much data do you need? What’s realistic for your scenarios? Do you honestly need 250GB of blog post content and comments? Are you trying to test blogs.msdn.microsoft.com? No? Then reel that desire for insane amounts of data back in a bit, please. Ensure you’ve got enough to meet your needs, but not too much more.
• What “shape” of data do you need? I’m sure the hard core data folks have a fancy term for this, but I’ve always used “shape” of data to describe patterns in how the data is laid out. “Shape” to me is a number of properties around how your data is constructed: how many users; users in what roles; types of content created by those users; the time period over which that content is created or your users interact with the system; etc., etc. Shape of your data becomes crucially important when you’re working with BI, reporting, or analysis testing.
• If shape is important, how long will your data be good for? You spent a lot of time this year building up a carefully crafted baseline dataset for your neat analysis tool. Will that dataset work as expected come 1 January of next year? Now all of a sudden you’re looking at a “Previous Year” window, not the “Current Year.” Ooops.
• Do we need real data or will dummy data be good enough? Working with geographic systems? Actuarial or financial systems? You’ll likely need something close to realism. Maybe you can get away with generated data, maybe not. Ask the question and get the answer.

You’ll notice all these considerations are extremely specific to the project you’re working on. There’s little or no chance you’ll be able to re-use anything about a baseline dataset from one project to another. Sorry.

Constructing Baseline Datasets

So you’ve carefully looked at the considerations above, and all the others unique to your project. You’ve got an idea of what you need, now how do you build it?

• Sanitize and reshape live data. This is generally most applicable to load testing where you just need masses of data that’s close in shape to real data. Why not go get real data from one of your large customers and use that? You’ll likely need some form of non-disclosure agreement in place, and you’ll likely have to scrub out personal information, but that’s easily done with some SQL-fu.
• Use a data generation tool. Visual Studio’s Data Dude (or whatever the official marketing-speak name for it is) will look at a database’s schema and help you create data in amazing detail. Red Gate has their SQL Data Generator tool, and there are any number of other tools as well. Spend a little time researching and finding out what will work for your team, environment, and project.
• Use your automated tests. If you have a suitable suite of automated tests, use those! Ensure you’re not deleting any created data or dropping databases after each run, then simply set up a series of runs and have your tests create everything you need. Handy, that.
• Build some custom SQL. Extremely useful if you have critically sensitive shapes required for your data. You’ll have to hand-build your data to ensure you’re meeting your very specific needs.

Once you’ve got your data constructed, zip it up and get it in your source control system. Yes, store this huge binary file in your source control. It’s a critical project asset, and will likely be tied to specific releases of your software. Treat this asset with the same respect you’re treating other project artifacts.

Wrapping Up

By all means, keep your tests granular, self-reliant, and have them handle their own setup/teardown where feasible.

That said, there are situations where a baseline dataset makes extreme sense. Keep an eye open for those situations and make use of baseline datasets to help save your sanity and deliver better software

31 Days of Testing—Day 17: Rules for Effective Data-Driven Tests

Updated: Index to all posts in this series is here!

Today’s post comes from guest author Seth Petry-Johnson who’s posted it at his blog. Seth contacted me when I started this series and expressed interest in writing about keeping database tests in good shape. That topic is music to my ears, so I took him up on his offer.

Seth will be following up his post with more at his site, so I encourage you to keep an eye on his blog too!

---

Tests that hit the database are slow, fragile, and difficult to automate. All the cool kids are using mocking and stubbing and in-memory databases to keep their unit tests isolated and fast, and that's awesome. I do that too (though my "cool kid" status is debatable).

However, there are times when talking to a real database is necessary. Maybe you're testing actual data access logic, or maybe you're writing some high end integration/acceptance tests, or maybe you're just working in an architecture that doesn't let you mock/stub/inject your way to isolated bliss. If any of that sounds familiar, then this post is for you!

Below is a list of strategies and suggestions for effective data testing that I've collected from years of experience testing large, "enterprisey", data-driven applications. Data tests will never be painless, but following these rules makes it suck less.

Rules for good data tests

1. Tests should create their own scenario data; never assume it already exists. Magic row IDs kill kittens!
2. Make liberal use of data helper and scenario setup classes.
3. Don't use your data access layer to test your data access layer.
4. Tests should make no permanent changes to the database - leave no data behind!

Rule 1: Tests should create their own data

One of the worst things you can do in a data test is to assume that some record (a customer, an order, etc) exists that fulfills your scenario requirements. This is a cardinal sin for many reasons:

1. It's extremely fragile; databases change over time, and tests that rely on pre-existing data often break (causing false-negative test failures).
2. It obscures the test's purpose. A test's setup communicates the data context in which our assertions are valid. If you omit that setup logic, you make it hard for other programmers to understand the scenario that you are testing.
3. It's not maintainable; other programmers won't know what makes customer ID 5 appropriate for one test and customer ID 7 appropriate for another. Once a test like this breaks, it tends to stay broken or get deleted.

In other words: relying on pre-existing data means your tests will break often, are painful to maintain when they do break, and don't clearly justify why another program should spend time fixing them.

The solution is simple: each test should create each and every test record it will rely on. If that sounds like a lot of work, it can be.... but keep reading to see how to keep it manageable.

Rule 2: Liberal use of data helper and scenario setup classes

Setting up the supporting data for a test sucks. It's time consuming and generally results in a lot of duplication, which then reduces test maintainability and readability. Test code is real code and should be kept DRY like anything else!

I've found it useful to create two different types of helper classes to attack this problem:

• Data helpers are utility classes that expose methods for quickly creating entities in the system. These classes:
  • Are generally static, for convenience.
  • Exist in the test project, not the main data access project.
  • Create a single object (or object graph), such as a Customer or an Order with its OrderItem children.
  • Create data with meaningful defaults, but allow the important fields to be explicitly specified where needed. (Optional parameters in .NET 4 FTW!)
• Scenario objects (aka "fixtures") represent specific data scenarios that might apply to multiple tests, such as the scenario in which a Customer has placed an Order and one of the items is backordered. These classes:
  • Exist in the test project.
  • Have public properties that identify key data in the scenario (e.g. the Customer ID, Order ID, and backordered Item ID).
  • Are instantiated by a test, at which time the scenario data is created.

In short, data helpers are low-level utilities for creating a specific data record in a specific state, while scenario classes represent larger contexts consisting of multiple entities. I have found that while the time needed to create these objects is not trivial, it quickly pays off as new tests are easier and easier to write.

Rule 3: Don't use your DAL to test your DAL

Tests for DAL code generally set up some data in the database, invoke the DAL, and then verify that the database was properly modified. I've generally found it difficult to use the primary DAL to quickly and concisely verify those assertions.

In some cases, the primary DAL may not expose a suitable API for doing record-level verification. For example, when there are significant differences between the logical schema exposed through the domain layer and the physical schema of the database, it may be impossible (or at least difficult) to write low-level data assertions.

In other cases, especially early in development, using the DAL to test the DAL creates dependency issues. For instance, many tests involve a sequence of events like "get entity by ID, save changes to it, then verify it was changed". If both the GetById() and Save() methods are currently under development then your test will give you inconclusive results until both methods are implemented.

In all of these cases I've found it valuable to verify data assertions using a LINQ to SQL data context. This provides a convenient, object-based representation of the data schema that is perfectly suited for verifying row-level operations were performed properly. This data context lives in the test project and is automatically regenerated (using SQLMetal.exe) whenever the schema changes, so it's a close-to-zero-effort solution.

You could also use a micro ORM like Massive, or anything else that makes it quick and easy to interact directly with the database.

Rule 4: Tests make no permanent changes to the database

Tests should be run often, and if you follow Rule #1 your tests create a lot of new data when they run. If you don't clean up after them, your test database will quickly grow in size. Also, if you point your test suite at the same database you use to run your app, you'll quickly get tired of seeing that test data accumulate in your views.

The easiest way to prevent this is to wrap each test in a database transaction, and then rollback that transaction at the end of the test. This performs the desired cleanup and also isolates tests running in parallel from interfering with each other's data.

There are a few different ways to approach this. Depending on your needs, check out this or this.

Conclusion

None of these techniques are particularly clever or game changing, but when used together they can significantly improve your data tests:

• When tests create their own scenario data, you don't need to run them against a particular known state. This reduces maintenance costs significantly.
• Investing in data helpers and scenario classes makes it easy to add new tests. The easier it is to write tests, the more likely that developers will actually do it.
• "Close to the metal" abstractions like LINQ to SQL make it easy to write row- and field-level assertions against the database.
• Adding some "auto rollback" behavior to your data tests keeps your database trim and tidy, no matter how many times you run your test suite.

Happy data testing!

31 Days of Testing-Day 16: Testing Web Services

Updated: Index to all posts in this series is here!

Today’s post is from long-time pal and one-time colleague Dan Hounshell. Dan’s been around the Heartland developer community for quite some time, although it’s been awhile since he’s gotten up at a conference or user group to share his great knowledge. (Yes, Dan, that’s a public Dope Slap for you to get presenting again!)

Dan is the Development lead for the Core at Telligent, makers of Telligent Enterprise and Community platforms for social networking. He’s spent several years building out the extensibility services layer for those platforms and helped build up several thousand integration tests hitting all aspects of testing the services. Dan’s post really speaks to the lessons he’s learned through this journey, which is why I was so happy when he agreed to write this post. He’s been there and done that.

In other words, he knows of what he speaks. (Well, or writes in this case.)

Testing Web Services

Let’s kick this off with a big dose of honesty. Creating tests for your web service layer is a lot of hard work. It involves working with a lot of moving parts that can be finicky and stubborn: HTTP, data, XML, maybe JSON, etc. At some point you may get very frustrated. At some point you may consider “skipping it just this once” or maybe even quitting entirely. Do not give up – the payoff is worth it. Once you have a good test suite in place new tests will be easy to add, you will get on a roll and eventually realize you have thousands of tests covering your application. You will be amazed and proud as you watch your test count grow from 100 to 500 and ultimately into the thousands.

The following tips and advice are based on knowledge that my team and I have gained over the last few years building out a hefty integration test suite for the web service layer of our rather large product.

Types of tests

I have seen a several different methods used for testing web services including unit tests, “light” integration tests, and “full” integration tests. The approach you choose will depend on your experience, how testable your application is, and your goals for your tests.

Unit testing is valuable if your web service layer has been built in a way that allows it to be tested in isolation. If you already have a thorough test suite of integration tests and functionality tests then unit tests may be all that you need. In this scenario you could unit test common methods, serializers, any mappers or converters, etc. Adding unit tests should prove that your web services work properly when given the right data but they will not guarantee that they are getting it.

By “light” integration tests I am referring to only testing a layer or so beneath the web services along with them rather than the full stack. Perhaps you have a model layer directly beneath your web services that you can disconnect from the data/persistence layer. Mocking or faking the data will allow you to fully test your web services along with the logic beneath them. This approach is a good option if you already have a full set of integration tests and do not want to spend unnecessary time duplicating testing of the full stack.

The approach that we adopted is to do “full” integration testing with our web services. Think of where web services live… on top of everything else. Testing the full stack in this manner is akin to doing a functional test but without having to worry about the UI. We continue unit testing pieces of concern but spend the bulk of our budget testing the entire stack. Most of the rest of this article relates to integration testing web services.

Every journey begins with one small step

Before you can have a great test suite you have to write the first test. Do it. Make it simple. Do it wrong, it doesn’t matter. Don’t set out to create a grand testing framework, write just one test in one test fixture on one page. New up a web client, make a request to an endpoint on your local development site that will return some data, and write a couple of asserts to make sure the response contains something expected. Now run your test(s). Rinse and repeat. Congratulations, you are on your way!

Refactor, refactor, refactor

Now that you have some tests in place start looking for duplicate/similar code. Make note of every time you cut and paste the same setup code from one test or test fixture to another. Start refactoring those sorts of things out into helpers or into base classes. There is no reason to have the same web client setup code in every test, refactor it so you’ll have something more like this:

var response = MakeGetRequest(baseUrl, “/api/users.xml”);

I am compelled to issue a warning at this point because this is a hole I’ve fallen into myself. Be careful about refactoring too far. Avoid so many layers of abstraction that you hinder test readability. Jim also mentioned this in a previous post, “Day 8: Pay Attention to Your Tests’ Setup!”.

In addition to refactoring the tests themselves, look at the code you are using to setup data for your tests and refactor that as well.

Standing up data

Build a library for standing up data that is easy to use with as little input as possible.

var user = test_data.CreateUser(“bob”);

In the above example passwords, email addresses and other things may be required to create a user in your application but you ought to be able to supply default values for those.

Add fluid interface support to your test data creation methods to make it easy and fast to create complex sets of related data.

var blog = test_data.CreateBlog().WithPosts(3).EachWithComments(3);

Compare that to the tediousness of the example below:

var blog = test_data.CreateBlog(); 

var post1 = test_data.CreateBlogPost(blog); 

var post2 = test_data.CreateBlogPost(blog); 

var post3 = test_data.CreateBlogPost(blog); 

var comment1_1 = test_data.CreateBlogPostComment(post1); 

var comment1_2 = test_data.CreateBlogPostComment(post1); 

… 5 more lines of the same go here … 

var comment3_3 = test_data.CreateBlogPostComment(post3);

Consider adding a session monitor to your test data creation library that will keep track of the objects that you create in order to delete them after your tests finish. This allows your developers to concentrate on testing the functionality and not concerning themselves with cleaning up test data when they are finished with it.

Providing data

In order to perform a full integration test of your web services you will more than likely need some data store to provide known test data or allow for creation of data on the fly. If your web services only serve up mostly static or known data, you are not allowing creation or updating of data, then you may be able to use your existing local development database for your test suite. Another option is to build a separate database with known data for use by tests, which will work fine for mostly static data or a small number of tests. However, the best option for dynamic data is to use a separate data store from your development/staging database(s). Creating a new database on the fly, filling it with data as-needed for each test, and destroying it when testing is completed is the cleanest method for testing but it is expensive and will increase the time it takes for your test suite to run.

Specifications

Do yourself and your team a favor by writing your tests using specification syntax. I’m not saying that you need to adopt TDD, BDD or the Talking Mister Ed but just to make your tests readable. Use an existing specification tool or develop your own wrapper around your chosen test framework’s Assert method. You will appreciate the naming standards when you have hundreds of files and thousands of tests in your suite. The below code is a simplified example of something you might see in our web services test suite:

Folder: Users

Filename: when_making_a_get_user_request_with_invalid_username

[TestFixture] 

public class when_making_a_get_user_request_with_invalid_username () 

{ 

    // setup stuff for http request goes here 

    [Test] 

    public void response_code_should_be_404()

    { 

        response.Code.ShouldBe(404); 

    }

    [Test] 

    public void response_should_have_errors()

    { 

        response.Errors.ShouldNotBeNull(); 

    }

    [Test] 

    public void response_should_have_user_not_found_error()

    { 

        response.Errors

                .Where(x => x.Message.Contains(“User not found”)) 

                .FirstOrDefault()

                .ShouldNotBeNull(); 

    }

}

The above is very readable and is a benefit to your developers, testers and product/project managers. Adopting a specification syntax allows your test suite to describe exactly what your web services deliver.

Foster the process

Building a good/thorough/dependable test suite is as much about making testing a first class citizen of the development process as it is writing and running tests. You must evangelize writing tests. To make sure that your test suite gets built and maintained you have to require tests with each code change and require that checks for those tests are part of the dev/code review process.

Be strict about feature development – updated code should include updated tests and updated documentation. For web services work I recommend adopting a workflow approach of

1. Development

2. Create tests (can be swapped with #1 if you TDD)

3. Create/Update documentation

4. Dev review / Code review (make sure tests and documentation are in place)

5. QA review

6. Build proof of concept, possibly throwaway app, that makes use of your new web service functionality

If you have good tests in place your QA review may be as simple as explaining the tests that were created and showing that they work.

Get everyone involved. Let everyone know “this is how we do things when we work on web services”. Empower team members to make wholesale changes, refactor, and become part of the solution.

Other tips

Test the edges. Make sure your web services are returning exceptions/errors when they should be.

Always ask “how can/should/do I test this” when creating/updating code. As Jim has mentioned in many of his previous posts in this series, testing should be part of the development cycle not something optional or tacked onto the end. How can you know that something works without testing?

Make writing tests fun. Try stuff that you would not/cannot use in production code.

Always look for new things to test and new ways to test.

“You can’t test everything. Get over it.”

Run the section of tests in the area you are working regularly, as in several times a coding session. Run the complete set (if possible) before committing your changes.

Enjoy!

Remember that building and maintaining a good test suite is a journey and not a destination. At some point you will realize that you have succeeded in building something great and that all the hard work is worth it. It is a pleasure watching your test suite grow over time and notify you of bugs introduced to your code base. Your customers will appreciate it too!

31 Days of Testing—Day 15: Cucumber is Not A QA Tool

Updated: Index to all posts in this series is here!

Today’s post is written by Tim Wingfield and cross-posted from his own blog. Tim’s a great pal who I had the pleasure (really!) of working with several years ago. Tim’s walked the walk around hard work getting great communication flowing with customers, and I’ve seen him succeed putting sensible, tailored workflows in to client environments.

Tim’s had some great experience figuring out what tools can help facilitate good communication, and when I shared my thoughts about this blog series he offered to write up some of his experiences and thinking around Cucumber. Without further ado, Tim’s nicely done post on why Cucumber isn’t just for QA.

Cucumber is Not A QA Tool

In my recent travels, and they have been plenty, I have come across some interesting views about Cucumber.

In one conversation someone was looking for "QA people with Cucumber experience." I thought that was pretty cool, they want some testing folks familiar with a rather popular test framework. Then came the second sentence, "I really don't know what cucumber does, but I know that QA people need to do it." (Yes, this person was a recruiter, if you couldn't tell by now.)

Contrast that with conversations with management and leadership types who don't know what cucumber is, but as you explain it you can see the words scrolling across their foreheads, "If we use this, I can lay off half my QA people! Cost savings!!"

Now, I can not totally discount both points, there is a little merit of truth in each one of them. In the first case, a QA group that knows and understands Cucumber can be a great asset to your product team and the long term quality of your product. Additionally, once Cucumber is in place it will free up some QA departments from doing repetitive manual tests and allow them to focus on more exploratory testing.

However, if you consider Cucumber to be only a QA tool, you are missing a good portion of what this and other Acceptance Test Driven Development (ATDD) or Behavior Driven Development (BDD) frameworks can provide.

Note: From this point forward I will use BDD in the discussion. BDD and ATDD are very closely related, almost indistinguishable in practice. I learned the phrase BDD first, so I tend to stick with that. And for writing purposes, it is one character shorter, thus promoting my natural laziness.

Collaboration

The biggest benefit to employing Cucumber, or any BDD framework, is that it tears down a lot of the communication walls to which we have become accustomed. Specifically the Gherkin syntax of Given/When/Then helps bring down those walls by writing acceptance criteria in a plain text format that everybody can agree on.

From the business's standpoint, finally being able to communicate to the development and QA teams in a clear manner has to be beyond welcome. Being able to work with multiple team members and arrive at something for the delivery team that says:

Given the customer has an item in their shopping cart
When the click the checkout button
Then I will have more money in my bank account

No more reams of documentation around a simple feature with wire frames, UML diagrams, feature documents, and other items buried in a wiki somewhere that gets ignored. Plain text is easy, and in my experience the easier something is to use, the better chance you have of it getting used.

Now from the QA person's standpoint, we have actual acceptance criteria that let us know if a feature is complete or not. No more vague language that allows room for interpretation where I can inadvertently add more functionality to a feature because it is what I think the user might want someday. Maybe.

Finally from the developer's standpoint we have features with an end point and a better description than, "Customer checkout feature." And not only do we have more clear, concise feature descriptions, when we are done writing code for the features we have a set of tests that will execute against the plain text features.

Build What We Want

Cucumber benefits all aspects of the business, but possibly its greatest benefit is that it allows us to easily agree to what "done" means for any given feature. The definition of done for a feature won't change over the life of the project, or if it does change we will have a broken Cucumber spec to alert us that we have changed our mind somewhere. But by having that agreement of "done" in place up front it frees up the developers and testers to focus on specifics of implementation and verification of the feature without ambiguity.

Developers of many languages keep a set of unit tests handy that will break if they make a change to existing code. It may or may not be an intentional break, but the point is they have a safety net to catch a logic change. A suite of Cucumber tests has that same watchful eye over your application as a whole, rather than just at the class or method level. If the business changes how a feature behaves and a cucumber test goes red, then the collaboration can kick back in across the whole team and the team can decide how to handle the original intent of the feature with this new feature request.

In short, we have a living record of what "done" has meant to every feature we have developed over the life of the project. By executing tests against those acceptance criteria, we also have a constant check that "done" for existing features remains done.

Cucumber Helps With Quality

So while I don't think Cucumber is a QA specific tool, it helps with overall product quality. By providing a good place for collaboration and by helping the team define "done" early in the feature life cycle, quality has become a first class citizen to your team. Quality is a whole team goal, not just members of the QA team. (By definition, people on the QA team assure quality, they don't build it.)

If you are not using Cucumber or another BDD framework, I would highly recommend looking into using one. However, look into those tools for the benefits they can bring your whole team not simply as a QA tool. And while they will help alleviate some manual work for the QA folks on your team, Cucumber is no replacement for a human being who can dive into the application in ways you never thought of when laying out the acceptance criteria.

After all who would you rather have exploring your app and poking around looking for things to go wrong?

Your QA team?

Or your users?

31 Days of Testing—Day 14: Tests as Specifications

Updated: Index to all posts in this series is here!

Your automated tests, be they unit, integration, or functional, are amazing things. They validate parts of your system are working as expected; they free you from tedious, repetitive manual testing so you can focus on high-value exploratory testing; etc., etc. etc. Automated tests are great.

What they aren’t always good at, in their basic form, is providing a good communication channel between the technical folks, testers, and business owners/stakeholders/folks who are writing the checks for your work. Take a look at any of the Selenium examples I’ve used in previous posts. Can you really expect your stakeholders to understand what’s going on? Should you? Likely not, but isn’t this a missed opportunity to get some great communication around what you, your devs, and your stakeholder actually want the system to do?

To steal a phrase from the mobile world, there’s an app for that. Well, not an app, specifically, but a broad set of tools, frameworks, and nifty software that can help you greatly facilitate those conversations.

Your non-dev folks on your team likely can’t understand something like the cascading drop down menu test I wrote up the other day:

[Test]

public void Working_with_no_content()

{

    string make = "Acura";

    string model = "Integra";

    string color = "Sea Green";

    browser.Navigate().GoToUrl(

    "http://www.asp.net/ajaxLibrary/AjaxControlToolkitSampleSite/CascadingDropDown/CascadingDropDown.aspx");

    var listOfMakes = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList1"));

    WebDriverWait wait = new WebDriverWait(browser, TimeSpan.FromSeconds(10));

    wait.Until<IWebElement>((d) =>

    {

    return d.FindElement(By.XPath(

        "id('ctl00_SampleContent_DropDownList1')/option[text()='"+make+"']"));

    });

    var makeOptions = new SelectElement(listOfMakes);

    makeOptions.SelectByText(make);

    var listOfModels = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList2"));

    wait.Until<IWebElement>((d) =>

    {

    return d.FindElement(By.XPath(

        "id('ctl00_SampleContent_DropDownList2')/option[text()='"+model+"']"));

    });

    var modelOptions = new SelectElement(listOfModels);

    modelOptions.SelectByText(model);

    var listOfColors = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList3"));

    wait.Until<IWebElement>((d) =>

    {

    return d.FindElement(By.XPath(

        "id('ctl00_SampleContent_DropDownList3')/option[text()='"+color+"']"));

    });

    var colorOptions = new SelectElement(listOfColors);

    colorOptions.SelectByText(color);

    browser.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));

    Assert.IsTrue(browser.FindElement(By.Id("ctl00_SampleContent_Label1")).Displayed);

}

but what they can understand is something like this:

Given I am at the main page

When I select Acura for Model

And I select Integra for Make

And I select Sea Green for Color

Then the result message shows ‘You selected a Sea Green Acura Integra!’

This idea of being able to specify your system’s behavior in plain English, or something close to it, is the fundamental concept of tools like Cucumber, JBehave, MSpec, and a host of other tools. Rather than losing behavior in the depths of C#, .NET, Ruby, or some other language we can bring it up to something in plain English like Cucumber, or a set of clearly understood tables in the case of Fitnesse.

Instead of me spending a lot of bandwidth trying to rehash all this, I am going to give you some reading assignments.

First, go grab a copy of Gojko Adzic’s Specifications By Example. He does an amazing job of taking the technology out of the picture and laying out a wonderful case for why you should look to specifications. He’s backed up his position with a number of great case studies. Perhaps one of the best books I’ve ever read on testing.

Secondly, go read Elisabeth Hendrickson’s awesome post on selecting test automation tools. She does a wonderful job of laying out how you might approach investigating and selecting some tools to help you bridge these communication gaps. The only additions I’d make to her great article is to say

1. I don’t think you necessarily need to write tests in the same language as your system. Figure out what tool works best for your team.
2. Think about your customer’s skills and teams if you’re handing off your system and tests. Will your customer be able to deal with all those Fitnesse or Cucumber fixtures? Just because you as a consultant think it’s cool and useful doesn’t mean it’s going to be great for your customer.

Finally, do your own trolling around for topics like Behavior Driven Development, or Acceptance Test Driven Development. There’s some wonderful stuff being written by wicked smart folks.

Tomorrow I have a post on using Cucumber in the real world written by my good pal Tim Wingfield. I think that will be a great practical follow on to this post!