Getting Selenium HTML Test Suites Running Automatically

We’ve been working on using Selenium to write tests around our major functional slices in Community Server. Selenium enables you to write a test script that opens up an instance of Firefox, Chrome, or Internet Explorer and drives it around your website. You can navigate, check functionality, and test for text on the screen. It’s an amazing tool, and it is leaps and bounds ahead of the web testing functionality in Visual Studio, IMO.

Selenium has a number of different parts to it: a server component to run the tests on a server, a remote control piece which lets you push tests through different browsers, and a grid piece to scale out execution for load testing/faster execution.

The Selenium server test runners support a number of different languages, so you can write tests in C#, Java, Ruby, or a number of others. Tests are written those languages, then compiled into an assembly, and executed in one fell swoop by the server which consolidates all the output from the tests into one individual results file.

We’re using program managers like myself to handle writing these tests, so we’re all over using Selenium IDE to record and maintain the tests. Selenium IDE records tests and stores them in an HTML file, so it’s easy for PMs to crack open and understand what’s going on.

Support for running HTML test “suites” in Selenium is pretty spotty, unfortunately. The Selenium team’s focus has really been on supporting real languages instead of HTML. (No, HTML is not a programming language! You aren’t writing code if you’re writing HTML!)  As a result, gaps in tools and a bug in the server code leave users with no way to automate a series of HTML test suites. You have to individually execute your test suites through Selenium IDE. While this is still tremendously useful, it’s painful if you want to run a large number of tests, or if you want to automate execution for dropping into a CI process like CruiseControl.NET.

Moving to the pure language environment isn’t what we want to do at Telligent for our feature testing. We want to keep the HTML test suites because our PMs all understand HTML and it’s easy for us to write and maintain tests – we don’t have to divert devs from focusing on their work. There’s an option in Selenium IDE to convert the HTML to C#/Rub/Java/etc., but that’s a one-way tool and can’t be automated. Too brittle, too labor intensive. Again, everything points our environment to HTML. Your environment may and likely does differ.

Dave Donaldson and I have been spending some time over the last couple weeks plinking away at getting this HTML suite issue solved, and we’ve finally gotten things up and working!  There are three steps involved:

0) Patch the Selenium server code to support HTML suites.

1) Build a script to execute your HTML test suites.

2) Build a script to read the suites’ output reports and consolidate the information for inclusion into your CC.NET report.

First off, you’ll need Selenium up and running on your server. I’m not going in to that in this post. The Selenium site’s instructions on setup can get you going.

You’ll also need a full version of the Java SDK on the server you’re running Selenium from, not just the JRE.

Next, you’ll need HTML Tidy. We use that to clean up some HTML goo from the Selenium test reports so that we can treat those reports as XML in PowerShell.

Lastly, you’ll need PowerShell.

Step 0: Patch Selenium

There’s a bug in Selenium RC that prevents the Selenium HTML test runner from correctly loading HTML test suite files. We need to fix that, so grab the Selenium RC source from the Subversion repository at http://svn.openqa.org/svn/selenium-rc.  You’ll need to apply the patch discussed here and then use Maven to rebuild the jar file. Maven will drop the updated file into the folder selenium-rc\trunk\selenium-server\target. The jar you’ll use is ‘selenium-server-1.0-SNAPSHOT-standalone.jar’.  You absolutely should rename that jarfile so it’s clear you’re running a patched version, not the stock one. Being a highly creative chap, I chose ‘selenium-server-1.0-SNAPSHOT-standalone-TelligentPatched.jar.’ Brings a tear to your eye, I’m sure.

 Step 1: Execute Your HTML Test Suites

(Note: Step 2: Deal With Reports is mixed in here…)

Launching an HTML test suite in Selenium is done by invoking Java to fire off the Selenium server (Dox on SeleniumHQ site). The snippet below shows some PowerShell parameters, i.e. “$browser”.

java -jar selenium-server-1.0-SNAPSHOT-standalone-TelligentPatched.jar 
        -htmlSuite 
	$browser 
	http://localhost/csfunctionaltests/ 
	$testSuitePath 
	$testReportPath

The “-htmlSuite” param says you’re running the test in an HTML suite as opposed to a C# assembly, Java jar, etc.  “$browser” lets you specify what browser to use. Chrome, Firefox, Opera, and Internet Explorer are among those currently supported. “http://localhost/csfunctionaltests/” is the root of the site your Selenium scripts will execute against. “$testSuitePath” is the test suite’s file location. “$testReportPath” is where the report for the specified test suite will get stored.

To start off with, we use a number of parameters when invoking this PowerShell script. This lets us keep the script the same but tweak its execution in different environments for our build server or local dev boxes. The parameter statement looks akin to

# Get params, most likely passed from a .bat file.
param($browser, $testScriptsRootDir, $reportDir, $tidyDir)

There are a number of ways to figure out what test suite files you need to run. We append “TestSuite” to all our suite filenames, i.e. “ForumCRUDP0TestSuite.html.” Using this convention, you could search a directory structure for all those test suites and iterate over them, launching the server jar as needed. Note that the test suite’s output report won’t get merged with other suite reports. You’ll need to have a separate name for each report and merge those after the run’s complete. Fear not, more details on that later.

For now, we’ve hardwired in our test suite locations in a predefined array:

# Setup pairs of test suites with their report names. 
#  Ex: ("myTestSuite.html", "myTestReport.html").
$suites = 
	("CommunityServer\Forums\P0\ForumCRUD\ForumCRUDP0TestSuite.html", 
	  "ForumCRUDP0.html"),
	("CommunityServer\Forums\P0\PostCRUD\PostCRUDP0TestSuite.html", 
	  "PostCRUDP0.html"),
	("CommunityServer\Wikis\P0\WikiCRUD\WikiCRUDP0TestSuite.html", 
	  "WikiCRUDP0.html")

Note that these aren’t full paths, they’re partial paths under a common root. We’ll use an input parameter to control where that common root is, enabling us to easily support different environments on our build box and local dev machines.

Next we set up some global variables for accumulating stats on each test suite:

$accumulatedTestCount = 0
$accumulatedTestPasses = 0
$accumulatedTestFails = 0

Now we can iterate through our list of test suites with a simple foreach. The loop sets up paths for the suite and reports, runs the suite, then does some processing on the output file to gather up statistics. We use HTML Tidy to clean up the report so we’re able to use PowerShell’s XML data type to easily pull important bits from the report.

This loop can use some serious refactoring, but hey, we’re just at the “Green” stage of things right now. Note that I’ve done some line breaking for readability here.

foreach ($testSuiteAndReport in $suites)
{
	$testSuite = $testSuiteAndReport[0]
	$testReport = $testSuiteAndReport[1]

        $testSuitePath = join-path $testScriptsRootDir $testSuite
	$testReportPath = join-path $reportDir $testReport
	
        "Running $testSuitePath..."
	$buildScriptDir = join-path $testScriptsRootDir "BuildScripts"
      
        set-location $buildScriptDir
	java -jar selenium-server-1.0-SNAPSHOT-standalone-TelligentPatched.jar 
		-htmlSuite 
		$browser 
		http://localhost/csfunctionaltests/ 
		$testSuitePath 
		$testReportPath
	
	"Cleaning up HTML..."
	set-location $tidyDir
	./tidy.exe -m -i 
		   --doctype omit 
		   --output-xml true 
		   --numeric-entities true 
		   $testReportPath
	
	$report = [xml] (get-content $testReportPath)

	#
	#Hardwired positions based on current Selenium HTML report
	#
	$currentSuite = 
		$report.SelectSingleNode("//table[@id='suiteTable']/tbody/tr[1]/td/b")
	
	$currentTestCount = $report.html.body.table[0].tr[2].td[1]
	$accumulatedTestCount += $currentTestCount

	$currentPasses = $report.html.body.table[0].tr[3].td[1]
	$accumulatedTestPasses += $currentPasses

	$currentFails = $report.html.body.table[0].tr[4].td[1]
	$accumulatedTestFails += $currentFails
	
	if ($currentFails -gt 0) 
	{
		"`nFailures present: " + $currentSuite.get_InnerText()
		$suitesWithFailures += $currentSuite.get_InnerText()
	}	
}

OK, so we’ve looped through our array of test suites to run each individual suite and save its report, we’ve cleaned up those reports, and extracted out various statistics from them.

Now we can directly work with PowerShell’s XML type to create the summary results we’ll hand off to CruiseControl for its reporting. (You do remember that’s what we started off with as our goal at the top of this post, right?)

First we’ll create our XML structure, then we’ll list our accumulated totals, following on with looping through test suites with failures, then finally closing out the XML structure and writing it to our output file.

"Building Selenium report..."

$xmlOut = "`n"
$xmlOut += "" + $accumulatedTestCount + "`n"
$xmlOut += "" + $accumulatedTestPasses+ "`n"
$xmlOut += "" + $accumulatedTestFails+ "`n"
$xmlOut += "`n"

"Suites with failures:"
foreach ($failure in $suitesWithFailures)
{
	$xmlOut += "" + $failure + "`n"
	"* " + $failure
}

$xmlOut += "`n"
$xmlOut += "`n"

$finalReport = join-path $reportDir "selenium-results.xml"

$xmlOut | out-file $finalReport

We’re finally through the entire script!

We run this script by invoking it from a batch file shown below. I’ve broken lines for readability, although the path to the PowerShell script will likely look like complete poo on my blog. Deal with it.

powershell.exe
	D:\Builds\CommunityServerFunctionalTests\Working\Trunk\Tests\FunctionalTests\BuildScripts\RunSeleniumTestSuites.ps1'" 
	-browser 
	"*firefox" 
	-testScriptsRootDir 
		"D:\Builds\CommunityServerFunctionalTests\Working\Trunk\Tests\FunctionalTests" 
	-reportDir 
		"D:\Builds\CommunityServerFunctionalTests\Artifacts" 
	-tidyDir 
		"D:\Downloads\HtmlTidy"

The entire PowerShell script can be found on my site here.

We use a simple XSL to merge the “selenium-results.xml” file into CruiseControl as well as a mail notice that goes out.  That XSL’s fairly simple:

<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">

    <xsl:output method="html"/>

    <xsl:template match="/">
        <xsl:if test="//seleniumReport">
            <br />

            <table border="0" width="100%">
                <tr>
                    <td>
                        <!-- Header table -->
                        <table width="100%">
                            <tr>
                                <td style="background-color: #000066; padding: 5px;">
                                    <span style="color: #ffffff; font-weight: bold; font-size: 12pt;">Selenium Summary</span>
                                </td>
                            </tr>
                        </table>

                        <!-- Stats table -->
                        <table>
                            <tr>
                                <td style="padding-left: 5px;">
                                    <span style="font-weight: bold;">Total Tests:</span>
                                </td>
                                <td style="padding-left: 5px;">
                                    <xsl:value-of select="//seleniumReport/totalTests" />
                                </td>
                            </tr>
                            <tr>
                                <td style="padding-left: 5px;">
                                    <span style="font-weight: bold;">Total Passed:</span>
                                </td>
                                <td style="padding-left: 5px;">
                                    <xsl:value-of select="//seleniumReport/totalPass" />
                                </td>
                            </tr>
                            <tr>
                                <td style="padding-left: 5px;">
                                    <span style="font-weight: bold;">Total Failed:</span>
                                </td>
                                <td style="padding-left: 5px;">
                                    <xsl:value-of select="//seleniumReport/totalFail" />
                                </td>
                            </tr>
                        </table>

                        <!-- Table to display failed test suites -->
                        <xsl:if test="//seleniumReport/failedSuites/failedSuite">
                            <xsl:variable name="failedSuite" select="//seleniumReport/failedSuites/failedSuite" />
                            <br/>
                            <table cellpadding="4" cellspacing="0">
                                <tr>
                                    <td style="border: 1px solid #649cc0; background-color: #a9d9f7;">
                                        <span style="font-weight: bold;">Failed Test Suite</span>
                                    </td>
                                </tr>

                                <xsl:for-each select="$failedSuite">
                                    <tr>
                                        <td style="border-left: 1px solid #649cc0; border-right: 1px solid #649cc0; border-bottom: 1px solid #649cc0;">
                                            <xsl:value-of select="." />
                                        </td>
                                    </tr>
                                </xsl:for-each>
                            </table>
                            <br />
                        </xsl:if>
                    </td>
                </tr>
            </table>
        </xsl:if>
    </xsl:template>
</xsl:stylesheet>

OK, that’s plenty to digest, and there’s lots of room for improvement in the process. As I said, we got past the Red stage and are at Green. Refactor comes as we figure out any hitches in this.

Feedback is welcome, and I’d especially love to hear from any Selenium folks who’ve also solved this problem!