QuickTest and TeamBuild for Continuous Integration

A while ago Grant posted about integrating QuickTest (QT) into a Team Build continuous integration server. I said I would post about my half of the process. I’ve have been a bit busy of late to organise all my notes and publish this post. However it’s all organised now so, without further stalling for time…

The first part of the process is to ensure that you have a good QuickTest project setup This is what I did and why:

Organisation of Projects

An automated functional testing project (known to QuickTest as a Test) in my opinion should represent a single system. All test cases that actively test this system should be stored in this project. This ensures that our tests are all stored together.

A project should have a name and to simplify the integration with TeamBuild the project should be called “quick-test”. The folder that the quick-test project is in should be named after the Branch in source-control.

Example:
-The test team branch
|-The Name of the System
| |-quick-test

The common naming conventions will help ensure a close correlation between the code that is used to build a system and the automation code used to test it. This is especially relevant if the QT code is in a different branch to the app-code.

Note: Here there were two primary ways to configure the projects within TFS. The first is to put the functional testing code in the same branch as the code that it tests. The unit tests are also in that branch structure and with the code and all related content are together.

The second method is to have a separate branch for the Testing Team and then under each, a folder for each project. For convenience sake we named the sub-folder the same name as the code branch. Under this was our quick-test folder. Load-runner or any other tool that tests this same project would also be stored at this level.

We went with the second method because QT repository files can get quite large and this would have been a problem for the developers on an already strained TFS server. I also understand that workspace mappings may alleviate this problem further. I’m not 100% here as SVN is my source control system of choice (don’t tell Grant).

From an access management perspective, giving testers, who will probably be unfamiliar with the processes around source control, access to only one branch may make their lives simpler.

Even though we went with the second method, I would investigate the first method for your organisation to see if it fits. It’s a better logical structure.

Organisation of Repositories
The repository is where the objects that are used by QuickTest to control your application are stored. There are three levels of repositories:

• Local – Attached to an asset
• Project – All objects for a project
• Global – Merge of all project repositories

I recommend not using a local repository as it is a maintenance nightmare. As you develop your automation test cases, continually push your objects into the project level repository and associate all actions with the project one. It’s much easier to maintain that duplicating objects into each local repository.

Furthermore I don’t recommend using a global repository either. Your objects should relate to the project you are testing. Once again this is primarily a hassle to maintain. If you have dedicated individuals managing the merging of object repositories then go for it. Otherwise give it a miss.

Now, with the current setup, projects are very isolated and it is not possible to write automated scenarios that interact across separate systems. For this I would suggest having a common project that is basically the same as a global repository. The difference being that it only contains the objects needed for cross system test cases are included in this branch. This involves less merging maintenance than merging all objects.

If you require a common repository, put it into a separate branch. This will make it easier for the build script to pull in the common repository when running a continuous integration instance.

The repository should be called repository.tsr in all scenarios. The location of the repository will determine whether it is project, global or common repository. Once again this makes it easier for the build script as it can expect a particular file rather than the name of the file being explicitly configured.

Actions and Quick Test

My definition of actions in QuickTest differs from standard usage. Personally I don’t like their setup and put some rules around when actions should be used. This in my opinion makes their usage cleaner.

An action to QuickTest is a single script that can be executed in isolation. To use, an action will represent either a single test scenario or a reusable test scenario. Scripts can call other actions. However, for the sake of simplicity down the track a script should only call reusable actions.

I’ll define some terminology to make this consistent:

• Action – a single script that represents a single test case. Like all good test cases it should have single objective. If you are testing more than one thing, write two actions. See my post on the action-test-scenario naming convention for information on succinct naming of test cases and reasons for singular testing scope.
• Reusable Action – An action in QT that you can be flagged to be “reusable”. To me this should not generate and pass or fail counts. It should merely ensure a state when called.

The first action of a project should be called zzRepository and we can use that to edit and mange our project repository in a local space. This action is deliberately excluded from CI results.

Reusable actions should start with the word ensure. They are required to establish a state on behalf of the calling action (they ensure a state). Once again these actions are not included in the CI results.


Back to Continuous Integration

Once you have set up your structure. Talk with your TFS guy to get the code checked in an in the right format. Once again this link to Grant's work will cover the next step.

In summary: the build script will pull in the launch QT pulling in the repository.tsr. If you use a common repository, pull that in as well. Then pipe in all actions one at a time to execute them.

After the build occurs QT produces a results.xml document. We need to take this and make it look even slightly better than the default QT one to make it useful. I really like the way CruiseControl.NET presents the NUnit xml format. So I wrote a xsl transform to make the QT -> NUnit transform.

Qt2NUnit.xsl

Finally you will need to adjust the CruiseControl.NET xsl for presenting NUnit xml-docs to reference TeamBuild images. You can use my version here