Maintainable and performant test suites are somewhat my addiction right now – simply because I’m finding more and more and more folks in every domain running in to the same sorts of problems: brittle, long-running test suites that end up sucking the life and morale out of teams.
Before you dive into a new project, or as you’re rethinking an approach to an existing one, take some time to consider what your test suite will evolve to look like. I’m not talking just unit tests, I’m talking your full stack of tests: unit, integration, functional, and perhaps even performance.
I’ve got a number of topics I’ll be writing on over a few posts, but this particular post centers on an easy win: keeping your test cases focused on what you really need to test. This means you need to cut out any unnecessary steps or UI goo that don’t pertain exactly to the functionality you’re testing.
In my Surviving Growing from Zero to 9,000 Selenium Tests talk at Rocky Mountain Ruby, I used an example of testing that User B can reply to a forum post from User A.
“As a user, I want to reply to another user’s forum post, so that I can create a conversation”
There are any number of things which don’t pertain to that test, including things like:
- Can User B log on to the system?
- Does User B’s name show up when she’s logged in?
- Can User B browse to User A’s forum post?
- Can User B properly format bold, italic, and hyperlink text in the reply?
Logging in to the system and browsing to User A’s original forum post might take you ten seconds, plus you need validation checks that User B actually did log on. If you have 800 test fixtures, and logging on and navigating to a point in your system takes 10 seconds for each test fixture, then you’re looking at 8,000 seconds just for that effort. 8,000 seconds / 60 seconds per minute gives 133-ish minutes.
That’s two hours of time spent just getting to the place where you need to actually do your test. Worse yet, if you have any problems with your logon or navigation functionality then you’re going to have false failures for those areas which masks potential failures in your post-a-reply functionality.
Look carefully at your system and see how you might be able to bypass those particular authentication and navigation issues. Maybe you can store a generated cookie for User B, enabling you to completely skip the logon process. Perhaps you can figure out how to navigate directly to User A’s forum post instead of using your system’s browse/search/whatever navigation.
Taking these steps can greatly reduce the brittle and slow nature of your tests.
Next let’s consider how you actually enter the text for that reply. Are you using a rich text or some other fancy editor? How does that editor render on your page? Is it locked in some iframe or other odd element that’s difficult to point your automation framework at?
Perhaps your system has an alternative editor that’s simpler to deal with. If that’s the case, look to use that simpler editor instead–you’ll find your tests for interacting with content become much cleaner and easier to maintain.
I’ve talked with developers who’ve created completely different configurations for their automated testing runs where entire slices of functionality get cut out in order to simplify testing of other discrete, controlled areas.
These sorts of concepts—cutting out unnecessary steps and eliminating irrelevant functionality—pay off in spades as you continue evolving your test suites. Look very hard at each test case as you’re fleshing it out and ensure you’re not biting off automation of steps you don’t need to.
Please note I’m not telling you to skip testing your logon and browsing functionality. (Or whatever it is you decide to cut out.) You’ll definitely need to test those bits and pieces, but you can do them in isolation instead of throughout the rest of your test suite.
Keep your test suites running as smoothly as possible by staying focused on exactly what you’re trying to test. Figure out ways to cut out all the remaining cruft. You’ll be much happier.
2 comments:
I understand what you're saying but it seems like a limited viewpoint - limited to a single test. When you take the big picture into account, isn't it better to have bigger test scripts which test more things, so you have fewer test scripts total? For automated stuff I can see the advantage of more discrete tests, but for something I need a human to do, I want the activity to be as efficient as possible, so wouldn't it be better to give them a script that tests 50 requirements, instead of 50 scripts that test one requirement each?
@jasmine: Yes, this post was very much focused on automation.
I agree with you that manual test scripts need to be very efficient; however, I'd make a subtle differentiation between test scripts and test /cases/. The cases need to be focused and carefully thought out, but you can certainly string a series of related ones together so you're covering as many requirements as possible while executing those great test /cases./
Small difference, but an important one.
(I'd also say that such a large script may point to a need for more automation so that your testers can do more productive exploratory testing while some tool bashes away on those other cases.)
Loved your comment, though. Thanks!
Post a Comment