Video of my Selenium Talk from Rocky Mountain Ruby Posted

Back in late August/early September I spoke at the Rocky Mountain Ruby conference in Boulder, Colorado. My talk was about lessons learned dealing with large functional test suites over my career. For example, at a previous job I ran a team that got up to 9,000 Selenium tests scattered across roughly 850 test fixtures. I’ve had similar experiences working on other projects too.

The talk was videotaped and is now up on the Confreaks video hosting site: Surviving Growing from Zero to 15,000 Selenium Tests. Yes, the talk’s title is wrong. I goofed when submitting it.

The fundamentals I lay out in this talk span all test tools and frameworks, so it doesn’t matter if you're writing your tests in Selenium or Watir, or using QTP or Visual Studio. Early in your automation effort you’ve got to address basic problems such as long-running suites, brittle tests, and focusing on automating only the most valuable, critical aspects of your system.

(I like to think that Telerik’s Test Studio helps you navigate these issues a little more easily but 0) I’m biased and B) you absolutely still need to use your grey matter and think about this stuff as you’re doing it!)

This talk was the genesis for my “Automation Isn’t Shiny Toys” talk I’m giving a number of times over the next few months. I’ll be writing up a number of blog posts around this both here at FrazzledDad and at my Telerik blog.

I’d also recommend you watch the Testing Panel discussion from the same conference. I sat on that panel with three other really smart folks and there’s a lot of tremendously useful information on it.

Upcoming Speaking Engagements

Updated AGAIN: I’m also giving my Intro to Unit Testing talk at the Cincinnati Financial User Group on 10/12.

Updated: I had the wrong date for speaking at Dayton! I’m doing that talk on 10/26!

I’m on the road a fair amount over the next couple months talking about maintainable automation and how to keep your sanity while moving forward with it. Here’s where I’ll be at between now and Thanksgiving. I’d love to see you if you’re in the neighborhood!

Name	Date	Location
StarWEST	10/6/11	Anaheim, CA
Cincinnati Financial User Group	10/12/11	Cincinnati, OH
DFW QA Association	10/18/11	Dallas, TX
Dayton .NET Developers Group	10/26/11	Dayton, OH
DevConnections	2-4 Nov	Las Vegas, NV
Agile Testing Days	11/15/11	Berlin, Germany

Three Book Reviews

Updated: Fixed the title which referenced two books when I’m actually reviewing three. Would have caught that off by one error had I written a test first…

Three long-overdue book reviews!

Debug It!

Debug It!: Find, Repair, and Prevent Bugs in Your Code by Paul Butcher, published by Pragmatic Press. ISBN 193435628X. This, coupled with David Agans’ Debugging, is a must-have on a programmer’s bookshelf.

Code goes wrong, systems get tetchy, and you need a logical approach to dealing with problems. Butcher’s book lays out lots of great tips and practices for finding, fixing, and preventing bugs in your software. I love his emphasis on empirical, measured approaches to troubleshooting, and I also like that he draws a clear border between finding and fixing bugs. You really do need to step back and take a breath after you’ve found a pesky bug, because you’ll want a different approach for fixing it and preventing other similar bugs.

“Debug It!” puts a great emphasis on controlling the environment and isolating one thing at a time as you work through finding bugs. Butcher’s sections making nondeterministic bugs deterministic and his logical steps for approaching bugs are wonderful. I also LOVE that he emphasizes, strongly, that the debugger is a tool of last resort. An important tool, but one to use only after you’ve exhausted other avenues. (I’m a firm believer the debugger is a time sink. Manually stepping through code is a horrible use of your time.)

Butcher covers a great many topics from the team’s culture to methodologies. Along the way he offers insight on tracking your bugs, practical tooling advice around assertions, and finishes up with a great section on Anti-Patterns. Here you’ll find some great reading around culture (had to work with a Prima Donna, anyone?), methodologies, and team culture (Code Ownership, FTW!).

This really was a tremendous book, and I highly recommend it!

The Manga Guide to the Universe

The Manga Guide to the Universe by Kenji Ishikawa, et. al. Published by No Starch Press. ISBN 1593272677.

I’ve gotten several Manga Guide to… books and they’ve all been wonderful. The Universe guide is no exception! The book’s comic-style is entertaining, although I could do without the weird females-as-sex-objects undertone, and the authors have done a great job bringing some really arcane, difficult topics to an understandable level.

The book weaves its education of the reader around a central plot where three girls are trying to write and create a school play. Their attempts to flesh out their storyline bring them in to contact with a number of other characters, each giving different insights into astrophysics, astronomy, and the historical evolution of science and theory in these domains.

I greatly enjoyed the historical aspect of the book – learning how scientists like Kepler, Galileo, and Copernicus made their discoveries is really fascinating. The book’s main storyline is interspersed with a lot of great sidebars on these practical matters in both historical and current contexts. You’ll read about how ancients attempted to measure distances of solar bodies, what dark matter is, how the universe was formed, and what scientists are working on currently.

This is another great addition to the Manga series, and I’m very happy to have gotten it!

The Agile Samurai

The Agile Samurai: How Agile Masters Deliver Great Software by Jonathan Rasmusson, Pragmatic Press. ISBN 1934356581.

I get many free books for review – this is one I paid for out of my own pocket and it was worth every penny. Clear, well-written, and extremely useful. This book stays very focused on what matters in building a successful agile practice without being preachy or vague.

Rasmusson doesn’t pull punches on silliness around “agile” being some silver bullet. He’s pragmatic and practical – and blunt when discussing schizophrenias like management by miracle (an unrealistic plan has a miracle event which results in working software. Yeah, that happens often!).

He’s also hard-nosed (in a gentle way) about the fundamental problem in any software project: poor communication. Consider this beauty from Chapter 3:

Q: What kills most projects?

A: The assumption of consensus where none exists.

Much of his book (well, so much of “agile,” really) is about ensuring clarity of communication.

Rasmusson walks you through a Project Inception workshop to “Get Everyone On the Bus” then moves on to the other standard agile topics. He does a wonderful job explaining user stories and estimation—one of the better jobs around points I’ve ever read—and he lays out a nice clear picture of why and how you should set up a visual workspace.

The book finishes up with a nicely done section on building the software. Rasmusson walks through the standard fare of test driven development, refactoring, technical debt, and continuous integration. If you’ve read much about agile you’ll likely not learn anything in this section, but its tone and content is every bit well done as the other sections.

I’ve read plenty of books on Agile. This one’s definitely in the top three.

Join NW Ohio .NET User Group for Their 10th Anniversary!

The great Northwest Ohio .NET User Group in Toledo is celebrating their 10th anniversary on Tuesday, 9/20. Richard Campbell will be presenting at their meeting, and rumor has it they’re having BBQ to top things off!

If you’re in the area I encourage you to head over and join them in celebrating their great milestone. Register at Eventbrite to help them keep an accurate headcount.

Maintainable Tests: Keeping Your Test Cases Focused

Maintainable and performant test suites are somewhat my addiction right now – simply because I’m finding more and more and more folks in every domain running in to the same sorts of problems: brittle, long-running test suites that end up sucking the life and morale out of teams.

Before you dive into a new project, or as you’re rethinking an approach to an existing one, take some time to consider what your test suite will evolve to look like. I’m not talking just unit tests, I’m talking your full stack of tests: unit, integration, functional, and perhaps even performance.

I’ve got a number of topics I’ll be writing on over a few posts, but this particular post centers on an easy win: keeping your test cases focused on what you really need to test. This means you need to cut out any unnecessary steps or UI goo that don’t pertain exactly to the functionality you’re testing.

In my Surviving Growing from Zero to 9,000 Selenium Tests talk at Rocky Mountain Ruby, I used an example of testing that User B can reply to a forum post from User A.

“As a user, I want to reply to another user’s forum post, so that I can create a conversation”

There are any number of things which don’t pertain to that test, including things like:

• Can User B log on to the system?
• Does User B’s name show up when she’s logged in?
• Can User B browse to User A’s forum post?
• Can User B properly format bold, italic, and hyperlink text in the reply?

Logging in to the system and browsing to User A’s original forum post might take you ten seconds, plus you need validation checks that User B actually did log on. If you have 800 test fixtures, and logging on and navigating to a point in your system takes 10 seconds for each test fixture, then you’re looking at 8,000 seconds just for that effort. 8,000 seconds / 60 seconds per minute gives 133-ish minutes.

That’s two hours of time spent just getting to the place where you need to actually do your test. Worse yet, if you have any problems with your logon or navigation functionality then you’re going to have false failures for those areas which masks potential failures in your post-a-reply functionality.

Look carefully at your system and see how you might be able to bypass those particular authentication and navigation issues. Maybe you can store a generated cookie for User B, enabling you to completely skip the logon process. Perhaps you can figure out how to navigate directly to User A’s forum post instead of using your system’s browse/search/whatever navigation.

Taking these steps can greatly reduce the brittle and slow nature of your tests.

Next let’s consider how you actually enter the text for that reply. Are you using a rich text or some other fancy editor? How does that editor render on your page? Is it locked in some iframe or other odd element that’s difficult to point your automation framework at?

Perhaps your system has an alternative editor that’s simpler to deal with.  If that’s the case, look to use that simpler editor instead–you’ll find your tests for interacting with content become much cleaner and easier to maintain.

I’ve talked with developers who’ve created completely different configurations for their automated testing runs where entire slices of functionality get cut out in order to simplify testing of other discrete, controlled areas.

These sorts of concepts—cutting out unnecessary steps and eliminating irrelevant functionality—pay off in spades as you continue evolving your test suites. Look very hard at each test case as you’re fleshing it out and ensure you’re not biting off automation of steps you don’t need to.

Please note I’m not telling you to skip testing your logon and browsing functionality. (Or whatever it is you decide to cut out.) You’ll definitely need to test those bits and pieces, but you can do them in isolation instead of throughout the rest of your test suite.

Keep your test suites running as smoothly as possible by staying focused on exactly what you’re trying to test. Figure out ways to cut out all the remaining cruft. You’ll be much happier.

Slides From Effective Distributed Teams Talk (Philly Day of Agile)

I’ve posted the slides from my Effective Distributed Teams talk at the Philadelphia Day of Agile up to SpeakerDeck.

Feedback on this session was extremely polarized. A few attendees felt I spent too much time covering forming the teams and what tools to look in to versus walking through specific problems around processes with distributed teams. That’s highly useful feedback and I’m considering how to possibly refactor part of the talk.

The other side of the feedback coin was extremely satisfying – there was a group of ten or so folks in the session who were highly interactive. I love these kinds of sessions where the audience dives in to the discussion with some questions and things they’ve been through. I got some wonderful input from a couple different attendees on how to get past cultural and language barriers, plus folks were great about sharing their own experiences with distributed teams.

I greatly enjoyed the chance to speak with the group, and I hope most folks got something out of it!

Rocky Mountain Ruby Conference Wrapup

Where to start? My great bosses at Telerik fully embrace some of my quirky ideas about diving in to different, new communities for us. As a result, I found myself right in the midst of a crowd of passionate Rubyists at the Rocky Mountain Ruby conference in Boulder, Colorado.

I was fortunate enough to get picked up for a session – I spoke in front of the crowd about my experiences being part of a team growing a Selenium test suite from zero to 9,000 tests. I also participated in a follow-on panel discussing testing as well as a panel discussing diversity in the software engineering industry.

I loved the format of the conference: a mix of Unconference, lightning talks, and sessions of either 20 or 30 minutes in duration. The overall conference is also single-track, so you’re able to avoid stressing about running around and deciding what sessions you’re going to miss.

My goal going to RMR was to chat up folks around long-running test suites and brittle tests. I had good fortune to get great discussions with folks I already knew and folks I’d just met. Here’s a surprise: long-running tests are an issue in the Ruby domain, too.  (Say it quietly, but they’re a problem in every domain…) I didn’t get any magic answers about solving the problem, but then I didn’t really expect to. What I did come away with were contacts to continue more detailed discussions later on.

The rest of the conference was also very rewarding. Every session I sat through was great, but there were several standouts for me.
•    Michael Feathers’ keynote. I’m sure he hears it all the time, but his Legacy Code book was extraordinarily influential for how I think about software. It was great to finally hear him speak.
•    Anthony Eden’s talk on building great APIs was tremendous. Work hard on making your APIs clear, concise, and simple. I’m hoping Anthony will submit this to CodeMash because it’s absolutely applicable to you regardless whether you’re writing Ruby, C#, Java, or Perl. Might not apply to assembly folks.
•    Jeff Casmir’s session on rescue projects hit home for a lot of reasons. The things he brought up rang very true with my own experiences, and I loved how he put out that there are many hidden costs for failed projects beyond just the financial bottom line. Rescuing a project is hard, hard work, but it can bring a set of tremendous personal rewards.
•    Justin Searls and Cory Flanigan’s talk on testing Javascript was really well presented. Paired presentations often fall flat. Justin and Cory had a unique, entertaining style that I enjoyed. (Because as a presenter myself I know how much work they put in to it.)
•    Avdi Grimm’s Exceptions talk re-re-reminded me that your team needs to sit down and hash out an exception handling strategy early on in your project. His talk was more on the mechanics of dealing with Exceptions and it was really need to see some intriguing ways you can deal with Exceptions in Ruby. I also had a great hallway chat with Avdi about development and testing in the embedded world. He corrected some of my misperceptions (based on previous work experience) about automated testing in the embedded world.
•    Mike Gerhard’s meditation talk/exercise was a wonderful way to start off the conference. He encouraged folks to do short meditations during the week and Tweet about them with the hashtag #devmed (which is developer meditation, not developer medication). I’ve used short meditation off and on to help me regain some balance during extremely stressful times in previous jobs. Mike’s session reminded me of other great benefits of meditation – you’ll be seeing #devmed Tweets from me starting next week.

The culture and atmosphere of the conference were extraordinary. I was really upset about being gimped up – I missed a lot of hiking and trail running – but I hope to be back next year and make up for that!

Thanks to the organizers for picking up my talk, and thanks to all with whom I had great discussions!

Maybe You’re Not Right and They’re Not Wrong

It may be that you’re a bit too caught up in yourself if the organization you’re trying to force change upon isn’t willing to adopt that change. If you see yourself as the one voice in the wilderness trying to move a group in a new direction, then maybe you need to look at whether or not the change you’re trying to effect is for the organization’s good, or your own sense of self-worth.

Maybe that group or organization has had some successes. Maybe they’re getting things done in a way that’s different from what you’re passionate about. Does it make them wrong? Wrong in your view, perhaps, but perhaps not wrong for that organization.

Maybe your passionate cries for change aren’t being taken as a positive thing. Maybe those cries are being heard in a completely different fashion than you intend, regardless of how many different approaches you try. Maybe instead of forming allies in your cause, you’re alienating those around you. Maybe instead of being seen as a motivating agent of change, you’re being seen as a distraction or worse a hindrance.

Maybe the organization’s also using you as a token: “Yep, at least we’ve got <him/her> around to keep us straight!” Those words are easy for people to say. Implementing the changes you’re passionate about aren’t quite so easy – but again, maybe the things you’re so passionate about aren’t a great fit for that organization.

If the organization’s not doing things that put people’s lives in jeopardy, or if they’re not behaving illegally or unethically, then perhaps you need to re-think why it’s so important for you to make them change. Maybe they really don’t need to change to be successful in their own view – and that’s completely acceptable.

Maybe in that context you’re not right and they’re not wrong.

Move on. It’s OK. Find a place where you’re not the sole voice in the wilderness. You’ll be happier, they’ll be happier.

CodeMash Call for Submissions Opening 15 September

The CodeMash call for submissions will open up on 15 September. The window will be very short this year. Make sure you don’t miss out: Get ready for the 15th!

Follow CodeMash on Twitter, monitor the CodeMash site and news feed, and keep up here for further details.

Speaking at the Philly Day of Agile

I’m going to be speaking at the Philly Day of Agile on 10 September. I’ll be giving my talk on making distributed teams work well. There’s also a lot of good content on making teams work well regardless of your location. I’m really excited because I’ve never been to Philly and I’m looking forward to meeting an entire new circle of folks from that community.

Phil’s herding the cats for this event and has a great summary of the lineup.

Please look me up if you’re attending! (Better yet, go register if you’re in that area and haven’t yet!)

IE 9 Security Settings on Server 2008

I do all my dev and testing work in a VM running Server 2008 R2. I inevitably run in to problems with the various default security settings for IE9 on this OS. In today’s incident, I was unable to get IE9 to properly render HTML 5 controls on KendoUI’s demo site. This is a problem since I’m working with Test Studio to record a few tests and it uses IE9 for recording.

Having a highly secured browser on your production boxes is a Good Thing; however, for your dev or testing systems it’s nothing but pain.

Two quick things you can do to ensure you’re able to be as friction-free as possible (well, aside from using a real browser like Firefox or Chrome, that is):

• Add the sites you’re working with to the Local Intranet or Trusted Sites zone. Tools | Internet Options | Security | Click “Trusted Sites” | Sites | Add. You can use wildcards to add entire domains.
• Turn off Enhanced Security Configuration. Start Server Manager, click the server, then Configure IE ESC (in the Security Information section). Turn ESC off for both Admin and regular users.

WARNING: You should spend a bit of time understanding what the impacts of this are for your dev/test environment, and you should never, ever do this sort of stuff on a production system.

A picture’s worth a thousand words, ergo:

UPDATED: Fixed busted picture URL. Whoopsies.

Two Important Blogposts for Functional Test Writers

Adam Goucher wrote two great posts that really lay out some critical, foundational things around functional testing, particularly with Selenium. Go read both if you do any sort of functional testing. (Or want to.)

Implicit Waits – understanding how and why Selenium handles waits is important. This post is particularly focused on Adam’s Py.Saunter project, but the crux of the matter is important regardless of whether you’re using Selenium, Watir, or Test Studio. (Shameless plug.) Awesome money quote:

“One of the key benefits of doing functional automation is you, the script creator, gain a greater understanding of the application being automated”

The Great Locator Debate – it’s crucial you define your locators in one and only one spot. Adam walks through pros and cons of defining those locators Globally or Locally in your Page Object class. Money quote:

“The coupling of the locators with the Page Objects should be a tight one and the desire to break that model should be a trigger that perhaps your object model is not as complete as you thought it was.”

Go read both posts.

(And subscribe to Adam’s blog while you’re there…)

Considering Attending STARWEST? We have Discount Codes!

Are you considering attending STARWEST, one of the premiere software testing conferences? My employer Telerik has a discount code that can help you save a good amount of money off your tickets!

If you're interested please contact me via e-mail at work and I'd be happy to pass the code on -- no strings attached, and no, I won't add you to some mailing list! My e-mail is Jim DOT Holmes AT Telerik DOT com.

Cross-Posting With my “Work” Blog

A significant part of my new job at Telerik’ involves writing a fair amount of content for my blog at Telerik, plus creating regular webcasts.

I don’t want to cross-post everything from that blog here, but I do think there are a number of things I’ll be writing about which have applicability to folks who read this blog.

Moving forward I’ll be trying a number of things including posting full articles both here and “there,” or just posting simple announcements here (“I posted something on <xxxxx> at Telerik. Go read it if you’re interested.”)

My goal is to avoid making this blog seem like a shill spot; however, I believe there’s some good value to the things I will be posting over there.

As always, I’m happy to hear feedback from my readers. Let me know in the comments, via e-mail (see contact link on the sidebar), or via Twitter.

Updated: Thanks to the comment from Jay, I’ve updated my work blog with an RSS feed. Sorry for the oversight!

Renaming a System Running SQL Server

I do all my dev and testing work in VMWare virtual machines which I clone off of a baseline image. I rename those new images based on their usage, and I constantly forget how to handle renaming the SQL Server instance.

Ergo, in order to make it easier for me to find the info next time I do it: here is the MSDN article showing how to do this.

Want to Improve/Change Your Career? Get Hooked up in the Community!

Are you hesitant about getting involved with professional communities in your area? Get over that hesitation and dive in. Or at least get your foot in the water.

Community involvement rewards me in many ways: I’m around smart people who love to share their knowledge, I find new ways to solve problems, I get energized by those same people, etc., etc. (I also get lots of great bacon at CodeMash.)

If that’s not enough for you, consider the simple fact of how involvement you’re your professional community can impact your career opportunities. All four of my last jobs have come directly because of my involvement with the Heartland community. Let me run those short stories by you so you can get a picture of just how my participation in the community helped.

I left the workforce in 2005 to be a stay at home dad. During this time I wanted to stay current with things in the .NET world and I got sick of driving to Columbus to their .NET group, so I started the Dayton .NET Developers Group. I worked with James Avery and kicked off the Dayton-Cincinnati Code Camp, one of the region’s first software conferences for any community. Because of this work (and my blogging and co-authoring Windows Developer Power Tools) I was awarded my MVP status from Microsoft. In the space of just a short year I’d grown a tremendous network of really smart folks who were passionate about the same things I was.

Community contact job 1: When it came time to re-enter the workforce in 2006 I spread the word I was looking for work via a blog post. Within two hours I had three contacts, one of which was my community pal Chris Woodruff who referred me in to NuSoft where I landed a position as a principal consultant.

Community contact job 2: When it became time to move on from NuSoft I reached out to my community pal Brian Prince who was running the Solutions Practice at Quick Solutions in Columbus. Brian had a great group of folks working from him who were all heavily involved in the Heartland’s developer community, and I knew it would be a wonderful opportunity. We met for lunch on Wednesday and on Friday I had a job offer in hand. Brian was the only person I reached out to for my search.

Community contact job 3: Life at Quick Solutions was terrific; however, it was a minimum of a three hour commute each day and that was just too much impact on my family life. I reached out to a couple community contacts, but my main lead was through two more Heartland community homies Dave Donaldson and Leon Gersing. They hooked me up with Telligent where I had a great run of various accomplishments. I can’t recall how many different people I spoke with about opportunities, but I am fairly sure it was no more than three or four. All were folks/companies I knew through my community network.

Community contact job 4: My new position at Telerik also came about through community contacts. I learned about the test tools division at Telerik by chatting with a few folks at a couple conferences, and I started looking in to the tools and frameworks. I reached out to one pal at Telerik and inquired about opportunities. One thing led to another and soon I was sleeping on the floor at Chicago’s O’Hare airport after having flown down to Austin for a face-to-face interview with the testing tools folks. (No, Telerik didn’t make me stay overnight in an airport; serious weather issues diverted me to an epic journey to get home in time for work the next day.)

---

There you go. My last four jobs, all through community contacts. Not once did I use Monster.com, Craigslist, or a recruiter.

Here are some specifics about why you as a job hunter should look to community contacts:

• Better chance of finding open opportunities. You’ll hear about neat opportunities before they’re posted on job boards, etc. Moreover, you may find companies willing to make openings for you if you’re a good fit for them.
• Better chance of getting past screens. Personal referrals count for a lot with companies, and your chances of getting past initial screens and actually landing an interview are much higher.
• Better chance of a good fit. Since you will know those community contacts, you’ll have a better understanding of the potential employer’s culture, mindset, and staff. That exposure over time is a much better picture than you can get during a short interview process.

Community involvement is a wonderful thing for oh so many reasons; however, if for no other reason, get involved so you have a better channel to protect and advance your career. The bright, passionate, engaged folks you really want to work around are all deep in to the community. Hook up with them. It will help you and your career.

Follow Up to Maintainable Automation

During my first-ever webinar with Telerik this week I spoke briefly about learning how to keep your tests maintainable. This is something near and dear to my heart – I’ve been on a number of projects where we’ve had to suffer through test suites which became ever more brittle as time went on.

Brittle tests break frequently due to unrelated changes in the system’s workflow, UI, or business rules. Of course, tests should fail when something directly relating to the test is broken—that’s why we have the tests!—but one change to a web page unrelated to the specific area you’re testing shouldn’t completely derail that test. Brittle tests duplicate workflows and UI locators across many tests, forcing you to change hundreds or even thousands of tests when one webpage changes its layout. Brittle tests are overly complicated and have too many complex setup steps.

Brittle tests are the testing world’s equivalent of technical debt, shortcuts taken during development which end up causing you grief further down the road. Eventually the project team ends up spending a large amount of time simply cleaning up broken tests—and getting less and less work done around delivering value to the team’s customers or stakeholders. This can even lead to the team abandoning test automation completely, or worse.

Roy Osherove opens up his tremendous book “The Art of Unit Testing” with a very frank discussion of a project of his which failed because the team’s tests turned into a convoluted mess. Note they didn’t just abandon testing, the entire project failed. The team was spending more time trying to fix broken tests then they were delivering value to the customer, and the project tanked because of it.

Avoid this sort of drama by starting out your testing work with maintainability in mind from the beginning. Keep in mind the three major causes of brittleness in larger test suites:

• Large suites taking too long to run
• Changes to common workflow (such as logons) breaking multiple tests
• Changes to UI elements breaking multiple tests

Tests Taking too Long to Run

UI-driven tests will always be orders of magnitude slower than unit tests. As you reach hundreds or thousands of UI tests you may find your tests taking many hours to complete. You’ll find yourself unable to have regular, frequent execution passes of your functional tests, instead relying on everything to get run once a week over the weekend. Long-running test suites can lead to a mindset of ignoring the tests because feedback doesn’t happen until days later.

Solve this with two different approaches. First, look to scale out your test infrastructure. Get your tests running in parallel on multiple systems using some form of test runner agent. Selenium, in both its v1 and v2 incarnations, supports this through Selenium Grid, or you can look to splitting up tests via your own customized test runner. If you’re using Telerik’s Test Studio you’ll benefit from our combination of Scheduling and Execution Servers to do this for you.

Secondly, look to split out your large suites and pull out a minimal set of Basic Validation Tests (BVTs) to check only the most important set of features. These BVTs should check only fundamental operations like:

• Does the home page load
• Can a user log on
• Can a user create/retrieve/update/delete content

Using BVTs lets your team quickly execute a safety net of functional tests before making major commits, and you can also have these BVTs running on a much more frequent basis if you’re using some form of automated build or continuous integration. (And you should be!)

Use both of these approaches together to help deal with your long-running test suites.

Workflow Changes Breaking Multiple Tests

Your tests should focus on one discrete piece of your system’s functionality, i.e. does entering an invalid customer number display an error? However, your test will likely need to take other actions as a prerequisite—logging on to the system and navigating to the customer record area, for example. These prerequisites are likely common across many of your tests, and this can make them a significant risk for increasing complexity and brittleness.

These sorts of common actions need to be pulled out in to separate areas of your test suite so they’re defined once and only once. That way if your logon workflow changes (say you’ve added an additional requirement to input a value from a keyfob) you only need to update the workflow in one spot.

If you’re using Telerik’s Test Studio you can make use of our test-as-step feature to separate these common pieces out in to modules. If you’re working in Selenium, Watir, or some other testing framework then you can separate this commonality in to separate helper methods.

The main point of this is exactly the same as the software engineering concept of the Don’t Repeat Yourself (DRY) principle: Define functionality in one spot and one spot only.

UI Changes Breaking Multiple Tests

Changing UI can also wreak havoc with your tests. Element locator values (element IDs, XPath, CSS locators, etc.) will likely change if the UI is reorganized – and all your tests depend on being able to find those elements based on those locators. If you have your locators scattered through hundreds or even thousands of tests then you’re in for a sad time updating them after the UX team moved a widget from the right side of the page to the footer.

Mitigate this problem with the same concept I wrote about for workflow: Keep critical information in one and only one location in your codebase.  Use the same DRY principle for your locators.

If you’re using Selenium, Watir, or some other coding-based framework, then spend some time getting familiar with the Page Objects pattern. This idea has you create a separate class for each page you’re interacting with in your system. The locators are defined in each class, and the rest of your tests refer to or consume those page object classes. If something changes on the UI, you simply go back and modify that single page object class and all your other tests remain unaffected.

Telerik’s Test Studio helps with this as well by using our element repository. Elements you interact with are stored in one spot within Studio’s infrastructure. If an element’s locator changes it’s a simple thing to update that element to the new value – and you do it in one place only.

Wrapping Up

Automated tests are wonderful and I’ve been pitching (and practicing!) what I preach for years. That said, you have to approach your automation efforts with an eye to maintaining your suite over time. A long-term automation strategy isn’t just about writing great tests that help you deliver awesome software, it’s also about keeping your sanity as your software and tests evolve.

My New Job at Telerik!

I'm tremendously excited to announce my new career change: as of today I'm Telerik's newest evangelist!

I'll be spreading the word about Telerik's great Test Studio set of tools and their backing framework. Even more importantly, I'll be spending a great amount of time interacting with testers and developers across a number of communities! I'm really looking forward to being able to spend time talking with testers who are doing both manual and automated testing, and figuring out where we can help.

This new position really ties in to a number of things I'm passionate about: testing (of course), getting out in communities, interacting with folks to find out what works and what doesn't in their environments, speaking, writing, the list goes on. Test Studio speaks very dearly to me, having been through the wringer with leading a team writing a large suite of Selenium tests. The toolset offers up some really wonderful approaches to helping teams work through web and WPF automated testing -- and there's a LOT more coming down the pipe!

I'm very, VERY excited to be on board with Telerik. I'd interviewed with them five or so years ago, but chose a different path at that time. Thankfully they were still interested in me after all these years! Telerik's well-known for their community involvement; something that drew me to them years ago -- they were also the first sponsor of the Dayton .NET Developers Group, thanks to an introduction from Joe Kunk.

There will be a lot of exciting things going on, including me getting to spend a significant amount of time back doing technical work instead of leading a team and herding cats outside of that. I was extraordinarily fortunate to run a tremendous team of seven folks in my previous position; it's going to be equally as wonderful focusing my energies on creating content, helping educate folks, and best of all, LEARNING for myself.

Telerik has a great group of people working for them, and I'm honored to be among them!

Goodbye Telligent

Today’s my last day at Telligent.

I’m moving on to a very exciting position that hits a lot of my core passions, but I’ll leave that for a separate post come Monday. (And it will, shockingly, be made from a Mac… )

Telligent’s been a great place to hang my hat for the last 2.5 years, and management there was tremendously supportive in giving me free rein to build a QA team from scratch and run that team as I saw fit. While there were a number of things I wasn’t able to accomplish during my tenure at Telligent, I take great pride in having worked with a great group of testers who really transformed how Telligent ships software. We, as a great team, built something wonderful from scratch and that’s a tremendous accomplishment!

I’m also glad I’ve been able to gain experience working for a product company. I’d had years of experience working for consulting and contracting companies, but the environment of a product company is completely different. My experience here did nothing but cement up my firm beliefs in Lean: you simply can’t afford to build things only a few customers will ever care about. The pain and politics around trying to remove features you’ve released to the public isn’t something folks outside the product world will ever understand. Dodge all that pain and politics in the first place – avoid complexity, avoid building things the vast majority of your users won’t use frequently.

Many thanks to the folks at Telligent who put up with my oddball views coupled with a loud, often curmudgeonly personality. Most importantly, a load of emphatic thanks to the seven folks, onshore and off, who were on my QA team – you folks are some of the best people I’ve ever worked with and we did amazing things together!

A Mentor’s Worth: Breaking Things Down

Good mentors are worth their weight in gold for oh so many reasons: their experience gives them views into problems you’d never considered, their technical skills or domain knowledge lets them identify things you’d missed, and their longer careers may give them some calm perspective you’re too fired up to miss. They’re also exceptional at helping you step back from problems you’re too deep in to.

As I wrote recently, I’ve picked up guitar playing again and am loving how it’s stretching my thinking and learning in so many ways. I’m also finding great parallels between my experience as a mentor and how my guitar instructor guides me.

For instance, a couple weeks ago I was having serious difficulty on a piece around fingerpicking. It’s basic stuff, but well, I’m a beginner. I’d hit a total mental block and wasn’t able to get past some of the initial pieces.

John, my instructor, took out his pencil and quietly made a few marks on the page. “Do just these combinations first, forget the melody. Just focus on the chords.”

Wow. Well, wow and “duh!”

Stepping back and focusing on small pieces of the trouble is such a crucial problem solving skill. The problem was I couldn’t get myself to step back. I needed a guide to pull me back and show me the way. Break it down, simplify it.

This is the same sort of thing I really enjoy working with my mentorees/subordinates on. Got a problem? Let’s chat it up and see what it looks like. Then let’s step back and break it down. Dealing with a complex test matrix? Maybe we can figure out some common points, or look to some combinatorial pairwise magic. Got an overly complex, muddled test case? Let’s step back and think about what we’re really trying to test, then go after that test case with a scalpel to pare away everything that’s not essential.

It’s what mentors are great at: helping you step back and break large problems into manageable pieces.

Updated:Fixed busted image link.