Wednesday, December 28, 2011

31 Days of Testing—Day 21: Data Driving Your Functional Tests

Index to all posts in this series is here!

This post shouldn’t be confused with Seth’s awesome post he lent me on Rules for Effective Data-Driven Tests. He was talking about testing interactions with the database. This post will show you how to push sets of data through a functional test.

This post’s examples are in C# with Selenium. If you’d like to see how the same test rolls in Test Studio, please go check out the short video I recorded on Data Driving Dynamically Loaded Elements for Telerik TV.

I’m going to refer back to the post I wrote on day 13: Functional Test 201 (Common Problems), specifically, case 3 where the elements you are working with are loaded up in the DOM, but have no content. That example used the ASP.NET AJAX Cascading Drop Down site. Feel free to go explore the original post and AJAX site if you need to refresh yourselves. I’ll wait.

The example site I used offers up three option lists (make, model, color) and gives you a matrix of combinations to deal with. Writing up separate test scripts for each combination would be insanity, so let’s not. Instead, we’ll create a list of items to pass through one test, and iterate that test repeatedly through the list.

First off, I’ve refactored the original example from day 13 to make it more modular and readable. Day 13 worked for an elementary example, but let’s move on to something more real-world-ish. Here’s the crux of the new test:

 Test: Working_with_no_content_data_driven
   1: [Test]
   2: public void Working_with_no_content_data_driven()
   3: {
   4:     IList<Car> cars = CarFactory.Return_three_valid_cars();
   5:  
   6:     browser.Navigate().GoToUrl(
   7:         "http://www.asp.net/ajaxLibrary/AjaxControlToolkitSampleSite/CascadingDropDown/CascadingDropDown.aspx");
   8:     WebDriverWait wait = new WebDriverWait(browser, TimeSpan.FromSeconds(10));
   9:  
  10:     foreach (Car car in cars)
  11:     {
  12:         browser.Navigate().Refresh();
  13:     
  14:         Select_make(car, wait);
  15:         Select_model(car, wait);
  16:         Select_color(car, wait);
  17:         Validate_message(car, wait);
  18:     }
  19: }

First I’m using a factory to build a list of three cars. Here’s what that looks like:

 Class: CarFactory
   1: public static class CarFactory
   2: {
   3:     public static IList<Car> Return_three_valid_cars()
   4:     {
   5:         return new List<Car>
   6:         {
   7:             new Car { Make = "Acura", Model = "Integra", Color = "Sea Green",
   8:                 Message = "Sea Green Acura Integra" },
   9:             new Car { Make = "Audi", Model = "S4", Color = "Metallic", 
  10:                 Message = "Metallic Audi S4" },
  11:             new Car { Make = "BMW", Model = "7 series", Color = "Brown", 
  12:                 Message = "Brown BMW 7 series" }
  13:         };
  14:     }
  15: }

Here I’m simply creating a list of Cars right in the method. This factory method could just as easily reach out to a database, read from an Excel file, etc., etc. The point being, the test itself has no idea what the data source is—and that’s exactly how it should be! Hiding the data source in the Factory lets me change the source as needed without impacting any of the tests which rely on that Factory.

Here’s the inspiring Car class:

 Class: Car
   1: public class Car
   2: {
   3:     public string Make { get; set; }
   4:     public string Model { get; set; }
   5:     public string Color { get; set; }
   6:     public string Message { get; set; }
   7: }

Back to the actual test now!

Lines 6-8 navigate to the site and set up our wait, quite similarly to day 13’s example.

Lines 10-18 let us iterate through our list of Cars. If you’re working in Python, Java, Ruby, or some other platform then obviously things will look different. The idea is we loop through our cars and run the same test each time.

Note that line 12 explicitly refreshes the browser each time through. Because we’re pulling data back from service calls, I’ve found the page DOM can hold old contents around. Refreshing each iteration ensures I have exactly the DOM I expect to work with.

Line 14, Select_make() calls a newly extracted method to interact with the page’s drop down for Make.

   1: private void Select_make(Car car, WebDriverWait wait)
   2: {
   3:     var listOfMakes = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList1"));
   4:     wait.Until<IWebElement>((d) =>
   5:     {
   6:         return d.FindElement(By.XPath(
   7:                                       "id('ctl00_SampleContent_DropDownList1')/option[text()='" +
   8:                                       car.Make + "']"));
   9:     });
  10:     var makeOptions = new SelectElement(listOfMakes);
  11:     makeOptions.SelectByText(car.Make);
  12: }

This works exactly the same as described in the previous post: get the drop down list, wait until its contents populate with the desired make for this iteration. I’m passing in the Car class here, which as I’m writing this makes me realize I should instead be passing in only the Make property from the Car, not the entire Car. Select_make shouldn’t have to know how to deal with a Car, only with what it expects to.

Enough ponderings on software design for now. I’ll refactor later.

The calls to Select_model() and Select_color() work in exactly the same fashion:

   1: private void Select_model(Car car, WebDriverWait wait)
   2: {
   3:     var listOfModels = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList2"));
   4:     wait.Until<IWebElement>((d) =>
   5:     {
   6:         return d.FindElement(By.XPath(
   7:                                       "id('ctl00_SampleContent_DropDownList2')/option[text()='" +
   8:                                       car.Model + "']"));
   9:     });
  10:     var modelOptions = new SelectElement(listOfModels);
  11:     modelOptions.SelectByText(car.Model);
  12: }
  13:  
  14: private void Select_color(Car car, WebDriverWait wait)
  15: {
  16:     var listOfColors = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList3"));
  17:     wait.Until<IWebElement>((d) =>
  18:     {
  19:         return d.FindElement(By.XPath(
  20:                                       "id('ctl00_SampleContent_DropDownList3')/option[text()='" +
  21:                                       car.Color + "']"));
  22:     });
  23:     var colorOptions = new SelectElement(listOfColors);
  24:     colorOptions.SelectByText(car.Color);
  25: }

Now for the validation which checks that the expected message is correctly displayed:

   1: private void Validate_message(Car car, WebDriverWait wait)
   2: {
   3:     var messageActual = wait.Until<IWebElement>((d) =>
   4:     {
   5:         return d.FindElement(By.XPath(
   6:                                       "id('ctl00_SampleContent_Label1')[contains(.,'" +
   7:                                       car.Message + "')]"));
   8:     });
   9:     Assert.IsTrue(messageActual.Text.Contains(car.Message), "Message: " +
  10:                                                             messageActual.Text);
  11: }

The XPath in this method uses the “contains” function to check contents under the element pointed to by id ct100_SampleContent_Label1. I don’t check for an exact match—I only want to check that the message property’s contents for the current car are somewhere in that element.

There you have it: a simple data driven example for your functional test. Key takeaway: Construct your actual data list behind some sort of façade, be it a Factory or some other equivalent. Never let your tests themselves be responsible for constructing the data list. This ensures you’ll always have the easy flexibility to change how the data is built, where it’s built from, what it looks like, etc.

I should also point out that, as with all my examples, I’m not making use of the Page Object pattern. My examples here are all very linear with locators defined right in the tests. Why am I not using Page Objects for you to read? Two reasons. First, the examples are long enough and I’m trying to keep things fairly simple. Secondly, quite frankly I’ve not worked with it enough to be confident in showing you proper examples. Go do your own research on it, get to know it, and decide if it’s a sensible path for you to follow.

If you’re interested, you can find the complete source for this example (and the ones from Day 13) in my GitHub repository.

Tuesday, December 27, 2011

31 Days of Testing—Day 20: Refactoring a Monster Test, Part 2

Index to all posts in this series is here!

First off, I hope everyone’s had a great holiday break (or at least readers in areas where you got a holiday break!). I’ve been mostly offline since the 23rd and have greatly enjoyed the respite.

I left off the last post having pulled out the logon functionality into a separate method, plus gave it a bit of  robustness check to make it less brittle—the logon now checks to see if it actually needs to log on.

The logon test looks like this:

Test: Logon If Needed
   1: Navigate to : '/welcome'
   2: IF (Verify Exists 'LoginLinkLink') THEN
   3:    Click 'LoginLinkLink'
   4:    Enter text 'testuser' in 'UsernameText'
   5:    Enter text 'abc123' in 'PasswordPassword'
   6:    Click 'LoginButtonSubmit'
   7: ELSE

Our “main” test now looks like this:

Test: Retrieve a newly created user and validate user's data is correct
   1: Execute test 'Log In If Needed'
   2: Click 'NewContactLink'
   3: Connect to pop-up window : 'http://localhost:3000/contacts/new'
   4: Set 'ContactFirstNameText' text to 'New'
   5: Set 'ContactLastNameText' text to 'User'
   6: Set 'ContactEmailEmail' text to 'new.user@foo.com'
   7: Set 'ContactLinkedinProfileText' text to 'http://linkedin.com/newuser'
   8: Check 'ContactGovtContractCheckBox' to be 'True'
   9: Check 'ContactDodCheckBox' to be 'True'
  10: Check 'ContactOtherCheckBox' to be 'True'
  11: Desktop command: Drag &amp; Drop Neutral Lead Image to Lead Type Drop Target
  12: Click 'CommitSubmit'
  13: Wait for 'TextContent' 'Contains' 'New' on 'NewTableCell'
  14: Verify 'TextContent' 'Contains' 'User' on 'UserTableCell'
  15: Verify 'TextContent' 'Contains' 'new.user@foo.com' on 'NewUserFooTableCell'
  16: Verify 'TextContent' 'Contains' 'http://linkedin.com/newuser' on 'HttpLinkedinTableCell'
  17: Verify attribute 'alt' has 'Contains' value of 'Neutral' on 'New User Lead Type'
  18: Extract attribute 'href' on 'ViewContactLink' into DataBindVariable $(ContactLinkUrl)
  19: Coded Step: [Retrieve_a_newly_created_user_and_validate_users_data_is_correct_CodedStep1]
  20: Click 'ViewContactLink'
  21: Coded Step: [Retrieve_a_newly_created_user_and_validate_users_data_is_correct_CodedStep]
  22:             Connect to pop-up window : 'http://localhost:3000/contacts/7', ConnectToPopup=True
  23: Verify input 'ContactFirstNameText' value 'Exact' 'New'.
  24: Verify input 'ContactLastNameText' value 'Exact' 'User'.
  25: Verify attribute 'value' has 'Same' value of 'new.user@foo.com' on 'ContactEmailEmail'
  26: Verify input 'ContactLinkedinProfileText' value 'Exact' 'http://linkedin.com/newuser'.
  27: Verify attribute 'alt' has 'Same' value of 'Neutral' on 'LeadTypeImage'
  28: Close pop-up window : 'http://localhost:3000/contacts'

In this post I want to concentrate on getting the steps for creating the test contact out of this test and in to its own test/method. In these posts I’m using Test Studio for my examples, but the concept here is completely and absolutely the same regardless if you’re using Selenium, Watir, Visual Studio’s web test, or some other tool.

The eventual goal would be to have the actual creation of the user done via some form of backing API or service call. Getting the browser out of the business of creating your test data is a Good Thing, but it will likely be a multi-step process, especially if you’re new to automation and are just building up your team and toolset.

I’ve worked through this process a number of times, and sometimes the easiest thing is to first go ahead and get the browser doing those actions for you, but centralize the functionality so it’s only happening in one place. While this is slower and more brittle for test execution, it may be your only option at the start – in prior projects/jobs my automation team and I didn’t have skills and/or knowledge right away to start making system calls for doing this setup work. Leaving the browser to do the setup enabled us continue getting automation built as we learned more about the system, what we needed, and worked with other developers to gradually replace those browser-based setup pieces with real service calls.

For now, I’ll chop steps 2-17 in the above “main” test out to a new test. That test now looks like this:

Test: Create Test Contact
   1: Execute test 'Log in if needed'
   2: Click 'NewContactLink'
   3: Connect to pop-up window : 'http://localhost:3000/contacts/new'
   4: Enter text 'New' in 'ContactFirstNameText'
   5: Enter text 'User' in 'ContactLastNameText'
   6: Enter text 'new.user@foo.com' in 'ContactEmailEmail'
   7: Enter text 'http://linkedin.com/newuser' in 'ContactLinkedinProfileText'
   8: Check 'ContactGovtContractCheckBox' to be 'True'
   9: Check 'ContactDodCheckBox' to be 'True'
  10: Check 'ContactOtherCheckBox' to be 'True'
  11: Desktop command: Drag &amp; Drop Neutral Lead Image to Lead Type Drop Target
  12: Click 'CommitSubmit'
  13: Wait for 'TextContent' 'Contains' 'New' on 'NewTableCell'
  14: Verify 'TextContent' 'Contains' 'User' on 'UserTableCell'
  15: Verify 'TextContent' 'Contains' 'new.user@foo.com' on 'NewUserFooTableCell'
  16: Verify 'TextContent' 'Contains' 'http://linkedin.com/newuser' on 'HttpLinkedinTableCell'
  17: Verify attribute 'alt' has 'Contains' value of 'Neutral' on 'New User Lead Type'

Here’s what the main test now looks like after refactoring. I've also renamed a couple steps to make things clearer--something I hadn't paid enough attention to despite looking at this a number of times already!

Test: Retrieve a newly created user and validate user's data is correct
   1: Execute test 'Log in if needed'
   2: Execute test 'Create Test Contact'
   3: Extract attribute 'href' on 'ViewContactLink' into DataBindVariable $(ContactLinkUrl)
   4: [LogExtractedContactLinkUrl] : @"New Coded Step
   5: Open a pop up window so we can connect to it with correct URL
   6: [Retrieve_a_newly_created_user_and_validate_users_data_is_correct_CodedStep] : 
   7:     @"Connect to pop-up window for newly created contact", ConnectToPopup=true
   8: Verify input 'ContactFirstNameText' value 'Exact' 'New'.
   9: Verify input 'ContactLastNameText' value 'Exact' 'User'.
  10: Verify attribute 'value' has 'Same' value of 'new.user@foo.com' on 'ContactEmailEmail'
  11: Verify input 'ContactLinkedinProfileText' value 'Exact' 'http://linkedin.com/newuser'.
  12: Verify attribute 'alt' has 'Same' value of 'Neutral' on 'LeadTypeImage'
  13: Close pop-up window : 'http://localhost:3000/contacts'

OK, now we’ve got things nicely separated out. This test does its setup by logging on and creating a new contact. If I was in straight Selenium or Watir, I’d have those two steps consolidated in a fixture setup. If I was driving tests with Cucumber, Fitnesse, or something similar I’d again have those setup steps consolidated in their equivalent.

Now the overall main tests passes, but I’m still using the browser for setting up my test prerequisites. It works, but as I’ve repeatedly said, it’s slow and it’s brittle. Because I’ve moved that setup out to its own test, it’s a simple matter to replace the guts of that test with a call to some sort of a service or system call.

Don’t get all fancy and complex at the start. Look to the simplest thing that will work for you, even if it seems a bit clunky at first. Because my demo app is so trivial, I can go seriously low-tech for my solution here and simply call out to the command line to invoke SQLITE3.EXE to create my test user for me. In the Create Test User test above, I can completely replace steps 2-13 with one coded step:

   1: [CodedStep(@"Create Test Contact")]
   2: public void Create_Test_Contact_CodedStep()
   3: {
   4:     System.Diagnostics.Process proc = new System.Diagnostics.Process();
   5:     proc.StartInfo.FileName = @"d:\temp\sqlite3.exe";
   6:     proc.StartInfo.Arguments = "development.sqlite3 \"delete from contacts"+
   7:         "where email like '%foo.com';insert into contacts "+
   8:         "(first_name, last_name,email, linkedin_profile, lead_type) "+
   9:         "values "+
  10:         "('New', 'User', 'new.user@foo.com', 'http://linkedin.com/newuser','NEUTRAL');"+
  11:         "\"";
  12:     proc.StartInfo.WorkingDirectory = @"D:\projects\Telerik-Demo\db";
  13:     proc.Start();
  14:     proc.WaitForExit();
  15: }

Yes, there are a number of ugly things here: I’ve hard wired in paths to the executable, working directory, and database name. Certainly I could move these off to configuration settings for a more dynamic, flexible approach—and it would be a waste of time for this project. Take care with how far you go on these sorts of things. Be very Lean in your approach and only solve problems you need to.

If my system were more complex, the logical next step would be to move this from a command line call to a service or stored procedure. That would save me spawning up a separate Process, and would simplify the 15 lines above down to something like:

   1: [CodedStep(@"Create Test User")]
   2: public int Create_Test_Contact_CodedStep()
   3: {
   4:     return My_backing_framework.Create_test_contact("New", "User",
   5:         "new.user@foo.com", "http://linkedin.com/newuser", "NEUTRAL");
   6: }

(The returned value could very well be the database’s ID value of the newly created user. Get creative and use these as opportunities to do stuff that makes your testing easier.)

At my last job, my pals Dan, Jayme, and Jose were responsible for our web services extensibility endpoints. They spent a lot of time building up a framework to create their prerequisites. It was fairly tailored to their particular environment, but I knew it was something my testing group could leverage for our Selenium work. It took several months of part-time hacking around on all sides, but eventually we got their “test pack” setup framework playing nicely with our automation framework. It was a long, slow process, but well worth the effort—however, we didn’t start down that road until we were very certain we needed such an extensive supporting framework for us.

So there you have it. Step two in refactoring this test: getting setup steps out from your test, then moved off to a backing API or simple system calls. Do these moves gradually and always take the simplest steps possible.

In my next post I’ll wrap up this effort with some other thoughts about getting rid of “monster” tests.

Subscribe (RSS)

The Leadership Journey