Wednesday, June 18, 2008

Test Execution Problem in VS2008

A nasty bug slipped through the cracks in VS2008's test execution engine.  The problem is rooted in the whole test deployment concept in Visual Studio 2005 and 2008 where all assemblies are copied to a folder underneath the TestResults folder, then executed from there. 

I think this is a silly, overly complex way to execute tests, and it's particularly a problem in VS2008 because the test runner doesn't execute with that folder as its base path.  A bug in VS2008 causes the AppDomain.CurrentDomain.BaseDirectory property to be set to C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE instead of the proper subfolder under TestResults.

There's some good detail and a possible fix in this MSDN post.  Unfortunately, this doesn't work in my environment because the resolver's assembly path property value ("Public Assemblies;Private Assemblies") also gets appended to the path, further borking things up for me.

I'm moderately narked about the bug with the BaseDirectory slipping through, but I can understand it.  Bugs happen, they get fixed.  Hopefully there's a workaround for the bug while a real fix is being created.

I'm more narked about the underlying decision to have tests run from a location different than the project's build directory.  The overall concept is brittle and has too many moving parts.  I have to fool around with DeploymentItem attributes in code, I have to fool around with Deployment Item settings in the test configuration.

The idea of a TestResults folder is great if it were limited to, like, holding results. Reports, working files, etc., would all be appropriate for that folder.  Copying all the binary bits and test support files to that folder, then running everything from that target isn't the way a real unit test framework would do it.

Unfortunately, this bug is a complete show-stopper for me because I can't work around it.  The specific issue for my current project, based on VS2005, is a crapload of tests reliant on our NHibernate infrastructure.  NHibernate's Cfg.Configure() command, invoked with no parameters, attempts to load the hibernate.cfg.xml file from the current folder of execution.  Sure, you can specify a directory path to load that file from, but I'd have to do significant amounts of code ugliness to alter that input path based on production, test, dev environments.

I'm waiting on more help from Microsoft on this bug, but for now it means we can't proceed with moving up to VS2008 for this phase of the project.  I'm bummed.

5 comments:

  1. You can't just use the real testing framework you mentioned?

    ReplyDelete
  2. Surprised to see that Harman hasn't already posted telling you to use MbUnit. :)

    ReplyDelete
  3. @TheProkrammer, @Jon Kruger: I'd love to move to MbUnit; however, we've already got a significant amount of tests written in MSTest, so converting would be a problem. I'd think about investing the effort if the project were longer, but we're only looking at a few weeks' worth of work so we have to stay focused on knocking out features.

    ReplyDelete
  4. Jim,

    Did you ever find a resolution or work around for this?

    ReplyDelete
  5. Hi Jim,

    I'm with Ryan. Did you ever find a resolution or work around for this?

    Thanks;

    Dave

    ReplyDelete