Exploring Selenium 2: Introduction & Lessons Learned in Selenium 1

We use Selenium (version 1.x) at work as a regression / functional test tool. We’ve gone through a number of iterations over the last year and a half since I first started pimping Selenium and learned some valuable lessons. Now we’re in the process of migrating to Selenium 2/WebDriver and I’ve been really pleased with the results so far.

I thought I’d write a short series of short articles showing some of the things we’re learning about Selenium 2 / WebDriver.  To kick off the series I thought I’d lay out an overview of the journey we’ve taken so far, and some rationale why I knowingly chose a less-than-optimal solution at the start.

First, a quick overview of the bits and pieces of Selenium v1, the toolset we started out with:

• Selenium IDE (SIDE): A Firefox plug in that lets you record browser actions (navigation, clicks, input) and play back those. You can write assertions to check for things like text, elements, etc. Scripts can be saved out as HTML or exported to Java, Ruby, C#, and a few others. You can write your own templates for outputting scripts.
• Selenium Remote Control (Selenium RC): A Java-based server app that launches, manages, and kills browser sessions. You start up RC, then run tests written in one of the various supported languages (lots of client libraries around), and RC does the rest. RC lets you run your tests against different browser platforms. If your scripts are simple, you can run one test suite against Firefox, IE, and Chrome without re-writing. Theoretically…
• Selenium Core: This is the basic framework for both SIDE and RC. When you spin up RC you’re getting core as part of the environment.
• Selenium Grid: Manages splitting off test suites to run your tests in parallel, thereby massively speeding up your tests. (I haven’t used Grid.)

Initially we started out using Selenium IDE, the record and playback tool, for writing our tests. Some caveats should be noted about our test goals: we were building out automated test suite from ground zero and were focusing on very fundamental, simple functionality tests. Think CRUD operations, navigation confirmation, etc. No fancy UI manipulation, no dealing with Ajax or Javascript. With this in mind, we could use SIDE and save off the scripts as HTML files and feed those to RC. This approach had some pros and cons.

Pros:

• At the time, our Program Managers were our primary testers. SIDE’s record/playback made it easy to get some tests written. ( Some tests > no tests)
• HTML based scripts made it easy for folks writing tests to alter the scripts as needed because SIDE’s recording actions didn’t deal well with dynamic content or a number of other things. We’d have to clean all that up manually.
• RC takes HTML scripts as an input, so we could automate what tests we were able to come up with.
• HTML files are easily wrapped in to our source control system. Treat your tests like production code, because they are production code. Get your test files, regardless of format, in to your source control. Go on, I’ll wait here while you do it.

Cons:

• Record and playback tools are generally failures for anything above a trivial one-off script. The tools create fragile, wonky scripts which require a lot of tweaking to get right and more tweaking to maintain. (Yes, yes, yes, you may have had glorious success with tools from Rational or Mercury. 99% of the rest of the testing world can’t afford the hundreds of thousands of dollars for those tools.)
• SIDE and Selenium don’t do well with relative links in their HTML files. This made it impossible to centralize common functions like logging in to and out of the system, creating users, etc. This is perhaps the biggest strike – if ANY link changes anywhere, you’ve got to fix up every occurrence of it. Massive violation of DRY, massive fail.
• Managing Selenium RC’s server app during a build cycle can get tedious.
• RC turned out to be even more finicky with timing issues around scripts. Any web UI script is already very sensitive to timing issues around your page loading and rendering. HTML scripts were even more so.

I’d known about some of the SIDE limitations going in to the effort; however, I needed something that would be a quick win to show how functional tests could help the entire team. We had some great success with those initial tests, but I had always wanted to push to “real” Selenium tests via Ruby or .NET. I knew we’d have much better results, and it was also a way to get me back in to writing code on a regular basis. (Cue evil mastermind’s laugh.)

Making the case for moving to code for our Selenium was greatly simplified because I’d show the value prop of test automation already. We’d caught a number of regression bugs simply through the automation tests chugging away and finding things like “Whoops. The logon link isn’t on the main page anymore…” Sometimes you need to start really small and get a few victories on the board before pressing on to the better solution.

With all this in mind, I started writing Selenium tests using the .NET libraries and running those via RC. I started a migration path converting the old HTML tests to C#-based tests and passing those off to RC for execution. More pros and cons:

Pros:

• I get to write real code. I’ve been in a Practice Lead, Program Manager, or Quality Lead for several years now and haven’t been able to crack open Visual Studio but once every few months. Writing code makes me very happy, so quite frankly I’m not ashamed to list this as a pro.
• Using a “real” language to write tests instead of HTML scripts lets us start to build up a common library for frequently used calls. I can centralize things like log in / log out, content creation, etc. (I’m not centralizing tests for those steps, mind you, just the actions which might be prerequisites for other tests.) Building up infrastructure/an API for your test environment is critical to your success – speed, maintainability, readability, the benefits just go on and on.
• I can now use a languages tools like iterators, branches, etc. to start writing much more powerful tests and support functions.
• Did I mention I get to write real code again?
• Tests in native code run through RC are much, much more stable and performant. We saw a significant reduction in false failures and tests ran faster.

Cons:

• Your tests are now in real code. If you were hoping to leverage non-coders to help write your tests then you’ve cut off that resource. Now you’re looking at finding time from your dev team to write your tests, or you’re looking at finding QA resources who can write code well enough to write your tests. (I found a great one of the latter. No, you can’t have her.)
• Selenium RC is still in the picture. You have to spin it up, and you have to deal with false failures injected into your tests due to RC being the middle man broker.

At this point I’d been fortunate enough to get another couple full-time, dedicated QA resources on board. This gave me the flexibility to start looking at other tools. Based on a couple comments from Jeremy Miller and Adam Goucher I decided to look in to Selenium 2.

Selenium 2 is combining with WebDriver, a nifty pairing that lets you get rid of having to deal with the separate Selenium RC server and directly manipulate the browser. Less moving parts is a huge win in my book. The API is a lot cleaner, much more readable, and solves a number of technical issues around timing. The tooling is officially in “Alpha” status; however, it’s quite full-featured already and quite stable. We’ve been having great success with it so far.

I’ll follow this introductory post with several others walking through our move from Selenium 1 to Selenium2/WebDriver.

MSBuild: Targeting a Configuration from the Command Line

Update: Read the comments for some good tips from other readers.

Firing up Visual Studio to simply build a solution is nuts. Do it from the Visual Studio Command Prompt instead and save yourself time.

What I always forget, though, is how to specify a release build, ergo I’m writing this blog post so I can quickly find it the next time I need it.

From the command line, use the “/p:” switch and pass the value “Configuration=Release” like so:

msbuild CommunityServer.Sync.sln /p:Configuration=Release

I’m also in the habit of first doing a clean build before the release build to ensure I’ve got no leftover cruft:

msbuild CommunityServer.Sync.sln /target:clean

(Actually, I tie all this stuff together with SlickRun shortcuts and batch files, which is even better!)

Central Ohio Day of .NET is Soon!

This year’s installment of the Central Ohio Day of .NET is coming up soon! This year’s event is just a couple weeks away on 5 June. We’re returning to the handy Roberts Centre in Wilmington, nicely located equidistantly between Dayton, Columbus, and Cincinnati.

If you haven’t already, go have a look at the terrific session list, then go register.

We’ve had great turnout for each event – last year’s was over 200. If you don’t go you’ll be missing a terrific opportunity to improve your skills and network with your peers!

Wrap up From Ann Arbor DODN

I spent Saturday at the Ann Arbor Day of .NET. Jason, Scott, John, and everyone else involved once again put on a tremendous event. It’s one of the region’s best conferences and was absolutely worth driving up and back on Saturday.

I gave two talks and was flattered to be part of a productive lunch-time panel discussion on how one keeps abreast of the huge amount of technology that floods we geeks.

You can find the slides from my “Three Tips to Improve Your Development Process” talk here. Apologies to everyone who attended that – thanks very much for putting up with the system crash I had. (For those who weren’t there, PowerPoint completely, absolutely froze up. Rather than lose five or ten minutes restarting my entire system and trying to reset back to where I was, I just pressed on with the talk sans slides. I’ve given the talk enough times that I was comfortable with speaking to a few things I scratched out on the whiteboard.)

During that talk I made reference to a number of things:

• Shore and Warden’s The Art of Agile Development. Awesome book on so many levels. I’ve adapted my current retrospective format from it.
• Derby’s Agile Retrospectives. An entire book on retrospective goodness!
• Etherpad.com or Titanpad.com. Awesometastic online real-time collaboration tool for taking notes. I use this on a daily basis with my offshore folks.
• Twiddla.com. Online whiteboard. It’s awesome. I blogged earlier about doing retrospectives using this tool.

The deck for my “Leadership 101” talk isn’t really going to be a lot of interest to anyone. After all, it’s 14 slides total, one of which is blank, two being single pictures, one title slide, and the other slides having only one word apiece on them… The title slide has more words on it than are in the rest of the entire presentation. Go figure.

However, I did mention a few things I thought were useful for folks looking to learn more on leadership.

• The Leadership 101 series I wrote which fired me up to do this talk.
• D-Day from Stephen Ambrose. This, along with his work Citizen Soldiers, give you some amazing insight to the generation that fought WWII. You’ll learn about high-level leaders like Eisenhower and mid- and low-level leaders in the foxholes at the front lines.
• Band of Brothers, also by Ambrose. More amazing stories, this time centered on a company of soldiers who were in the entire European campaign.
• Ricardo Semler’s Maverick. Mind-blowing book about how Semler threw out everything most “experts” know about running business and instead empowered his workers to do awesome things.

Thanks to everyone who attended my talks. I hope you got something useful out of them!