Nice Acceptance Test Framework: Cucumber

I’ve spent some time over the last month fooling around with using Cucumber to write some acceptance tests for bits and pieces of the next version of Community Server. Cucumber’s great advantage is its lovely expressive grammar – a clear grammar for testing is critical to my view because it helps ensure developers, analysts, and the customer are all on the same page with how the system should behave.

For instance, we need to ensure a role has the appropriate permissions set for it. In this case, the Site Members role, by default, has 25 separate permissions assigned to it:

I could write up a Selenium test to navigate through all this and do individual assertElementPresent or some variant on each one of those, but that would be time consuming and frankly, the test report wouldn’t be all that clear, even if I used good names for the Selenium test suite and individual cases like I do for other tests:

Flip this around. Imagine that the person owning the feature can write a simple text file with contents like this:

Feature: Default Member Permissions in a Public Open Group

  In order to ensure the system is properly secured

  As a site administrator

  I want to check that roles are assigned 

    the correct default permissions for Public Open Groups

  Scenario: Check default permissions for the Members role

    Given I created the PermTest group as a Public Open group

    When I examine the Members role for the PermTest group

    Then the total number of Granted Permissions should be 25

    Then Granted Permissions should include Blog - Create Comments

    And Granted Permissions should include Blog - Read Posts

    And Granted Permissions should include Forum - Attach Files Locally

    And Granted Permissions should include Forum - Attach Files Remotely

    And Granted Permissions should include Forum - Create New Threads

    And Granted Permissions should include Forum - Post Replies

That snippet clearly states the goal of the Feature, plus it gives an exact scenario to test with a specific Given/When/Then set of conditions.

I can use Cucumber to run that clear language snippet through a framework and run tests against each of the “Then” conditions laid out above. (Code below) I’ll get a nice output with familiar colors (red, green) for the state of my conditions:

To me, this is just huge. I’ve got a specification written by someone who understands WHAT the system is supposed to do and HOW it’s supposed to behave. I can do a number of very cool things with that spec. First off, it can and should live in the source control repository where updates can be versioned. Secondly, I might be able to link to that item in source control from whatever project management tool I’m using – why duplicate that text in a project system when I can pull it straight in from Subversion, Git, etc. ? Finally, that expressive language is a thing of beauty for crossing the communication/understanding barriers between devs and feature owners/stakeholders.

My level of excitement around this is very, very high, in case you haven’t figured that out already.

Here’s the details of the implementation behind that spec above.  First off, you obviously need the Cucumber framework. Secondly, you’ll need Ruby.  There’s some C# examples in Cucumber, but I’ve not fooled with C# yet.

The spec above is written in plain text and saved as “MemberInAPublicOpenGroup.feature” in a unique folder.  We’re going to use Watir to handle this test, so in a folder called “support” we’ll need to have an “env.rb” file which is responsible for firing up our Watir IE instance and doing a bit of work:

require 'spec/expectations'

#require 'firewatir'

#Browser = FireWatir::Firefox

require 'watir'

Browser = Watir::IE

# "before all"

browser = Browser.new

Before do

  @browser = browser

  @browser.speed=:zippy #for Watir with IE only. Also try :fast

end

# "after all"

After do

  @browser.link(:text, "Log Out").click

  @browser.close

end

Note that I’ve got two lines referencing Firewatir commented out. I’m able to easily switch back and forth between browsers without altering my test code!

Cucumber expects the actual implementation code to follow that convention: step_definitions\<FeatureName>_steps.rb. We’ll write the actual test code in a file called “MemberInAPublicGroup_steps.rb.”

require 'spec/expectations'

Given /I created the PermTest group as a Public Open group/ do

    @browser.goto "http://localhost/csfunctionaltests"

    @browser.link(:text, "Sign in").click

    @browser.text_field(:id, "ctl00_bcr_ctl03_ctl07_username").set("admin")

    @browser.text_field(:id, "ctl00_bcr_ctl03_ctl07_password").set("pa$$word")

    @browser.button(:id, "ctl00_bcr_ctl03_ctl07_loginButton").click

    @browser.link(:text, "Control Panel").click

    @browser.link(:text, "Group Administration").click    

end

When 'I examine the (.*) role for the PermTest group' do |roleName|

    @browser.link(:text, "Dashboard").click

    @browser.link(:text, "Group Administration").click

    @browser.link(:xpath, "//div[@id='Common']/div[3]/div/table/tbody/tr/td[1]/div/table/tbody/tr[2]/td/strong/a").click

    @browser.span(:xpath, "//span[text()='PermTest']").click

    @browser.button(:xpath, "//input[@value='Edit']").click

    @browser.link(:text, "Permissions").click

    @browser.select_list(:id, "ctl00_ctl00_ctl00_OuterTaskRegion_TaskRegion_TaskRegion_TaskRegion_ctl02_ctl02_GroupRoles_RoleList").select(roleName)

end

Then /^the total number of Granted Permissions should be (\d+)$/ do |numGrantedPermissions|

    selectListItems = @browser.select_list(:id, "ctl00_ctl00_ctl00_OuterTaskRegion_TaskRegion_TaskRegion_TaskRegion_ctl02_ctl02_GroupRoles_listGrantedPermissions").getAllContents.length

    selectListItems.should == numGrantedPermissions.to_i

end

Then /^Granted Permissions should include (.*)$/ do |permissionName|

    @browser.element_by_xpath("//select[@id='ctl00_ctl00_ctl00_OuterTaskRegion_TaskRegion_TaskRegion_TaskRegion_ctl02_ctl02_GroupRoles_listGrantedPermissions']/option[text()='" +permissionName+ "']").should_not be_nil

end

The main parts here are the blocks for Given, When, and Then. Cucumber looks for those blocks, then grabs the regular expressions after those keywords. Matching blocks are executed and asserts/expectations are checked. Passing expectations means the item is written back to the screen in green; failures in red.

Keep something in mind here: This particular _step and feature is using Watir to test at the UI level. There’s no reason whatsoever that you couldn’t use this tool for testing at the integration or even unit level. Joe O’Brien, wicked smart guy driving EdgeCase around, said via Twitter that they’re using Cucumber at all sorts of different levels in their testing.

There are some cons to consider: you can still have an implementation disconnect if the dev writing the _steps file does something wrong. If you’re at the UI level then you’re also hostage to the pains of Selenium, Watir, Firewatir, etc. I continue to run into troubles with timing, but I’m working around those.

One tip: When you’re first starting out, add a line in your scenario which should fail. When you run the feature through Cucumber you ought to see a red line. If not, you’ve goofed up something in your step implementation. I learned this the hard way…

Use something like this

Scenario: Check default permissions for the Members role

  Given I created the PermTest group as a Public Open group

  When I examine the Members role for the PermTest group

  Then the total number of Granted Permissions should be 25

  Then Granted Permissions should include Blog - Create Comments

  And Granted Permissions should include Blog - Read Posts

...

  And Granted Permissions should include THIS SHOULD FAIL

to generate this:

This has saved me some time after some initial self-inflicted wounds.

I’m still evaluating this, and I’ve not wrapped it into any automated builds or CI reporting process. I’ll pass on updates as we continue to explore this.