Controlling Testing Environments

Why You Should Care?
Testing environments are fundamental to successful testing. The test environment is where testing occurs and without a controlled, regulated, stable testing environment you are undermining your entire testing foundation. Scary stuff!

What do I mean by controlling a testing environment? I mean ensuring:

• that you know that each environment has the correct code,
• that the various integrating applications have compatible versions,
• that the correct hardware and software configuration exists,
• that the data is legitimate and in the right quantities,
• access to the environment is restricted and,
• security policies mimic production

All of above items combine to make a stable, controlled, test environment.

Without proper management of testing environments whenever a defect is identified you have to:

1. identify the software build,
2. determine how long that build has been there,
3. determine if there is a later build available?
4. ensure that the date is valid?
5. review the hardware to ensure it matches production
6. review the additional software components to ensure it matches production

Beyond environmental stability there are particular test scenarios that you can now perform. You can engage in deployment testing. Every release the software package is released into production. How often is this software deployment process tested?

Other benefits are: when you receive a "bad" build you can un-install it and re-install the previous one until it gets fixed. Or, you can get two competing builds from the development team and compare them for performance. I am doing this one next week.


So how do we go about doing this?
The first step is to identify how many test environments you have / need. In summary, I like to see at least the following:

• Development - one per developer, usually the development box but ideally should be a VM or similar that matches production architecture/operating system/software configuration. Developers may call it a build box, but they do unit testing here, so it is a test environment.
• Development integration - one per project/release. Here the development team works on integrating their individual components together.
• Test - where the brunt of the tester's work is done. There should be a dedicated environment for each project.

The following environments can usually be shared between project teams depending on the number and types of projects being developed concurrently.

• User acceptance testing - can be done in other environments if the resources are not available. Ideally should be a dedicated environment that looks like prod + all code between now and project release. This is an optional environment in my opinion as there are lots of good places to do UAT and it really depends on the maturity of your project and your organisation's available infrastructure.
• Non-functional - performance, stress, load, robustness - should be identical infrastructure to production, the data requirements can exceed production quantities but must match it in authenticity.

More environments are possible. I didn't cover integration or release candidate environments (you may have duplicate environments or subsets for prod-1, prod and prod+1) and it really depends on the number of software products being developed concurrently. I won't be discussing the logistics of establishing test environments here nor how to acquire them cheaply.

To actually gain control. First talk to the development team about your requirements for a stable testing environment. Explain your reasons and get their support. The next step is not always necessary but can give you good piece of mind. Remove developer access to the test environments. I am talking about everywhere, web servers, databases, terminal services, virtual machines. If its apart of the testing environment they should stay out.

It isn't because you don't trust them. After deployment you probably shouldn't be on those machines either. Sure, there are some testing scenarios where getting into the nitty gritty is required, but not always and certainly not when testing from the user's perspective. The bottom line is that the less people who have access to these machines results in a smaller chance of accidental environmental comprise.


So what aspects do we control?
Primarily we need to control the Entry and Exit criteria to each environment. The first step is the development environment. Entry is entirely up to the developer and exit should be achieved when unit tests passed. As the next step is the development integration environment, the development lead should control code entry.

Entry into the test environment: regardless of the development methodology the delivery to test should be scheduled. Development completes a build that delivers "N chunks" of functionality. Unit tests have passed and they are good to go.

Developers should then prepare a deployment package (like they will for the eventual production release) and place it in a shared location that the deployment testers can access. It is now up to the deployment testers to deploy the code at the request of the project testing team (these are quite often the same team). Once a build has been deployed, some build verification tests are executed (preferably automated) and the testers can continue their work.

To move from test into any environment afterwards (release candidate integration, pre-production, etc) depends on the organisation but usually the following: Testing has been completed, defects resolved, user documentation produced and most importantly user sign-off has been acquired.

The final environments (pre-production, etc) are usually (should be) managed by a release manager who controls the entry and exit gates from each environment after test and on into production. I won't cover these here.


Evidence or it never happened!
Example A: About a month ago we had a problem where one of our test environments wasn't working as expected. It took the developer over a week to find the problem. Turns out another developer had promoted some code without letting anyone else know. The code didn't work and he left it there.

This could have been avoided if the developer didn't have access to deploy in our environment. Unfortunately he does, but it is something that we are working towards rectifying.

Example B: I once worked on a project that had five development teams. Two database groups and three code cutters. Had they been able to deploy when they wanted, our test environment would have been useless. None of the teams were ever ready at the same time and it would have meant we would have had code without appropriate database support. Components that were meant to integrate but did not match because the latest build of application x wasn't ready yet.

By waiting until all builds were ready and running through the deployment ourselves we ensured that our test environment was stable and had the same level of development progression all the way through.


Too much information, summarise before I stop caring!

1. Controlling Test Environments = Good
2. Focus on developing entry and exit criteria
3. Build up to production-like environments - each successive environment should be closer and closer to production.
4. Evolve towards the goal of environmental control rather than a big bang approach. Some transitions will take longer than others (i.e. getting the right hardware) so pick a level of control for each release, get everyone involved and implement it.
5. Get team buy in (developers, testers) - education is the key
6. Don't make the entry into the test environment documentation heavy.

It all looks too easy, how could this go wrong?
Get development buy-in. This is important you don't want to alienate the development team. Not all developers or development teams are inconsiderate, nor do they have ulterior motives. Usually it's a simple lack of awareness and discussing with them the direction you want to take with the testing environments will achieve two things. Firstly, they have greater visibility into the testing arena and secondly they often realise that they can help improve quality by doing less. Who doesn't like doing that?


Don't make it complicated: The goal of this is to achieve a high quality test environment to facilitate high quality testing. Don't produce a set of forms and a series of hoops that you need to force various developers and teams to fill out whilst jumping through. They won't like it and they probably won't like you.

When I first tried locking down an environment, I asked the developers to fill out a handover to test document that listed the build, implemented task items, resolved defects and similar items. I had buy in and for the first few cycles it worked ok. It wasn't great though. All I was doing was accumulating bits of paper and wasting their time by making them fill it out.

All I do these days is discuss with the developers the reasons why the environment needs to be locked down and to let me know when a new build is ready. I'm usually involved in iteration planning meetings so I know what is coming anyway. All that waffle they had to fill out is automatically generated from defect management, task management and source control software.

My testing environments are generally stable, developers are happy to hand me deployment packages and consider deployment defects just as important as normal defects. After all, deployment is the first chance a piece of software has to fail in production. It is also the first place user's will see your application.

It takes time to move towards a controlled environment and as you read in my examples, my employer is not there yet either, but we are getting closer.


One other note: You may not have the ability (whether technical or organisational) to perform development testing. See if you can organise to sit with the technical team that does deployments for you.

If the TV fits...

Every Wednesday I head over to my mum's place for dinner. Yesterday she gave me a TV. They had upgraded their two primary televisions to 32 inch LCD screens and didn't have room for the old ones.

It barely fits into my display unit and the image quality is much better than my old crappy tele. All I need to do now is find time to watch tv. I can only find enough time to watch 3-5 hours of tv a week which is usually watching my footy teams.

Blogger's Remorse

I really need to hold off posting at night. Even though it I spend a few days to a few weeks putting together my posts, I have them proof-read at least once and use a spellchecker, the tone of my posts is never the same after the fact.

I had to fix a few up from last night including renaming one post.

Blogging, like software development is a process of continual learning and evolution for me. I'm not where where I want to be regarding style, tone, spelling mistakes, but I am aware of where I am now and where I want to be. You can't get somewhere if you don't know where you are now.


I'll always leave my old posts up here because when I do achieve a writing style that I am happy with I can see how far I've come. I am starting to do this with defects as well. It would be nice to know how many defects I've caused in every application I've written since I first coded some basic in a Commodore 64. The type, cause, severity, etc, would also be useful to clear up the reporting. This way I could tell if my coding practices are improving.

I know TFS is starting to include this type of reporting through its Data Mining capabilities and testing tools like Quality Center also provide lots of reporting. If I can get a open-source setup working that I am happy with I'll post it here.

Without getting off topic too much more, let me wrap-up. Blogging is an evolutionary process where you get better by working on your craft. Just like software development. This is why my post here about analysing how you work to improve processes and therefore improve quality is just as relevant to blogging as it is to software development. Within a few weeks I should know whether just delaying a post until the next day when I can read it again will reduce the number of grammatical offences as well as improving the overall tone of each post.


p.s. Yes, I love statistics. I once had a spreadsheet that tracked the hours I slept, how I felt (subjectively of course), hours worked on various projects, productivity on each project and number of cups of coffee/water/tea I consumed per day in an effort to identify any correlations between them. Turns out sleep had the biggest factor on how I felt which directly translated into productivity.
p.p.s I'm not actually that remorseful over the posts, I just wish I had the ability to write with clarity on the day, not on the day after. Also RSS feeds don't reflect changes.

Bug: Facebook Advertising Application

Here is step two of registering for an advertisement on FaceBook:



and here is step three after a few clicks (select all users with relationship status):


What is the difference you may ask? In the second panel the user has selected each relationship status and according to the "people" count in the top of the page, the site is losing advertising reach of 1 million users.

EDIT: I changed this post as I didn't like the tone of it.

Usability: What is the ideal date control?

Mace is not the right answer, effective though. As exciting as date controls are they are pretty easy to get wrong. Date controls have the unfortunate problem of trying to represent a large period of time whilst trying to provide a good level of granularity over that time.

Users want to be able to specify a date and if the date control resembles something they are familiar with (i.e. a calendar) then even better. From what I’ve experienced being able to supply a date as quickly as possible doesn’t come into the equation with most non-technical users. That being said, users do not want to waste time trying to operate an unwieldy date control. They want to supply a value in as many steps as few steps they can conceive in their minds.

Developers and experienced computer operators (data entry personnel, power users) tend to want to do things as quickly as possible. This is at odds with user experience. This is also me. I can type over 120 words per minute. Most users can't. Remember that.

Application Platform
Where the application is running has a big impact on how the user is going to interact with the date control. Console applications obviously are going to use a keyboard mechanism. GUI Applications are a mixture of keyboard input and mouse control depending on how many fields need to be supplied. Web applications are mouse driven, and mobile/pda applications are pen driven (essentially mouse), arrow-navigation driven or touch pad.

The type of mouse also impacts usability of a date control. Touch pad mouses and nub mouses are a different user experiences than the standard hand-held mouse.

Date Selection
The types of dates the user selects is also important. How far in the past will the user be selecting a date? If it is a web-site where someone must be 18 years or older or an student records application that requires the date of birth, the date control is going to require the user select a year at least 18 years to something old like 30 (kidding, I mean 40… no 90.)

Are the dates in the past even allowed? Travel booking sites would rather you booked in the future. Does the user need to select a second date based on the first? Can the user supply a partial date? Is the user required to include a range of dates? Simple date range selection is non-trivial whilst keeping it simple, intuitive and aesthetic.


The Options
There are many examples of date controls out there, so I’ll try to provide a real live example of each one.

Drop Down City:

To be honest I had a worst example available but thankfully I can't seem to find it on google (if I can find it again I will post it as it was atrocious. Note to self: bookmark epic failures under "hindenburg"). To be honest it's one of the few times I have been glad to not find an example. A friend told me Immigration Australia had a drop down city control but I couldn't find it either. FYI Some aspects of their site are rubbish and I'll post that under at separate topic in a day or two once I collect my thoughts.


Above, the wordpress one isn't great. Personally I don't like it. Too many controls for something that could be done visually. Furthermore, the month selector doesn't always respond to user input forcing me into mouse input. Some may claim that this is a browser or OS issue. News Just In: The user does not care what you think, only what they experience.

Anyway, back to my original impressions: for starters, lots of drop downs look hideous. Secondly to select a specific date the mouse user has to click on a variety of drop down controls and then the keyboard user must supply some other values. A date control should support either all keyboard or all mouse and preferably both.

Single Box Date Control
Examples: (seriously, I had several examples planned. Perhaps people learn. I know I do so it's entirely plausible. When I find a relevant example again I will so post.). Anyway you have all seen it. The 12 character input control with '/' for dividers!

This option is the equal fastest date control for keyboard users and not usable at all for mouse users. However if you have a number of text fields before this date control then having a keyboard only field, I feel, is OK. The user will already have two hands on the keyboard at this point and it is only a jump away from the date control.

If the user has to take their hands off the mouse just to use the date control then this option is not a good choice.

This option is not aesthetically pleasing unless there are similar looking input controls near by.

Triple Box Date Control
The other fastest date control. Depending on the look and feel of the user interface this option can be aesthetically pleasing. It can also be ugly [here]. This date control is functionally similar to the single box control, or a completed entry should skip the user to the next field. If your users can specify partial dates then this option beats a single box hands down. It provides the easiest way of allowing the user to select which aspect of the date they may wish to supply (month for instance) whilst still displaying that the other two fields are empty.

One caveat with the three box date control is that you will need to make it clear which box is for the month and which is for the day. US date controls tend to be month first while day first is common in countries like the UK and Australia.


Calendar
The most visually appealing date control, the calendar gives the user a date selection control akin to the calendar on their desk or on the wall in their garage. This is a good thing. Your average non-computer literate user can work a calendar with ease. Unless implemented correctly a calendar is also a nightmare for date selection within a few months or a few years.

This website (Dynarch) has a calendar control on the right hand side. It works fairly well, its an odd concept to work in even years on the right, odd years on the left. Personally I don't like it. To find 2012 I have to go to 2013 and then back a year. Too much work, I would rather scroll than have to click on one side of dialogue and then click on the other side to fine-tune my search.

Still kudos go to them, because the calendar control is trying. My only other criticism of Dynarch is the word "Today" is in the middle of the control, it implies that the current date selected is today, like it is on my physical desktop calendar. It doesn't mean that. It means go back to today. This threw me off. With anything there is a usability vs education issue, personally that text should give a short hand notation of today. Quicklinks should be included below the control with other concepts like "next week", "next month". Depending on your content (always, of course).


To make a calendar usable for larger date ranges the following features need to be supported:

• Click on the month to activate a dropdown control to select a month. The month should open on June/July so that there is minimal distance to move to the desired month.
  
• Click on the year should allow the user to control the year more easily. I see couple of different options implemented here.
• Up Arrow / Down Arrow – this is a bad idea. Your date control should already contain double arrows for going forward or back one year
• Enter a year, this is useful if the year is a long way away from the current year
• A year slide. Looks like a drop down menu with about 5-10 years visible above and below the current year. At the top and bottom are two buttons for scrolling. These should be on-mouse-over activate or click-and-hold activated.

Other usability requirements that are a must:

• Let the user know that if you click on the month your can select a different month. The same applies to the year. Users rely on your visual feedback to let them know what is wrong and what is right within the confines of their education. Step outside those bounds and you lose. I've done it, you've done it. Don't be ashamed, realise a mistake and move towards the user.
  

What isn’t good for calendars? Scroll bars are a bad idea. It is far too difficult to find a specific year using a scroll bar. On one application I tested each scroll click moved 50 years.

Here is an example of someone that is pretty darn close: http://www.basicdatepicker.com/
They're date control only misses the need for auto-scroll one year, and supply forward one year, back one year options.

Some Other Thoughts
Do you need a date control? Seriously. For the beta registration page for No Horizons we used an age attribute. We don't really care about the specifics of someones age as long we could gather their age. This is very valid for websites where someone must be 18 years or over? Seriously, making someone work a serious of dropdowns and input controls to state they're 18 is ridiculous. A simple "I am N years age" where N is a supplied value is just as effective. To memorise: understand your demographics.

Much like piracy "cautionary messages" the only people you hurt are the people who do the right thing. Pirates leave off such messages when duplicated films and people who enter your site underage, etc, are lying.

Conclusion
So what is the best date control? It really depends on the application environment, whether dates are optional, but generally you should try to provide both, the input box input as well as the calendar. A set of input boxes for the keyboard users, this is especially relevant if you have a number of input fields that the user can supply. For visual applications and especially web or mobile applications you should provide calendar functionality.

Bonus Content: Date Ranges
Asking the user to specify date ranges is a non-trivial task. Often I am expected to supply a start and an end date. This is fine if I know the specific range I am searching. If I don’t then it’s trial and error until I do. A better solution is to provide that functionality for users that do know the date range as well as functionality for those that think in terms that users are familiar with. "It was last week" or "I am pretty sure it was in January" are concepts the user understands. Hell, as a developer/tester it's a concept that I think in.

Being able to search one to four weeks ago, and then in terms of months or years is more ideal. Show me all emails sent between June and August last year. That’s closer to how users think and it’s a lot easier to do than, show me all emails from the 01/06/2007 till the 31/08/2008. It’s a subtle difference but one the users prefer.

How would you implement this? You can allow partial dates in your input boxes for keyboard users. Secondly, you can let people double click on the month header to default to the first of that month and then close the window. Furthermore, quick options, like “last week”, “this week”, “this month” are handy shortcuts that make life just a little bit easier.

Usability Case Study: Facebook Advertising

A short case study on usability. It is all of a sudden relevant to me (because everyone is doing it wrong) and because I myself have learnt more about usability over the past six weeks than I thought I would. Therefore I feel I have an obligation to share my knowledge with you.

The other day I was investigating Facebook’s social advertising mechanism. Half of it was business related, the other half curiosity. I clicked the "advertise" link and was taken to a screen that pitched their advertising. After scanning that I clicked the “create ad now” button.

This took me to stage 1 of 4 of creating an ad.



This is also where I started to groan. All I wanted to know was how much it would cost to advertise on Facebook. I am not ready to start advertising, I am not even convinced that Facebook advertising is even a viable alternative.

The last step is called “Set Budget”. So I figured I could skip past the first three steps to find out. No Dice! I had to supply my URL, name and a brief description before I could continue.

Page 2 wanted my expected audience and the third, images and the advertising text. Luckily you can supply an image later, but dummy values in the advertising text were rejected as illegal words. At this point I quit. Seriously I said, fuck off.

Some advice to people who are looking to solicit services via the Internet. I don’t care who you are or what you think you know about what I want. You don’t. Secondly, I am not going to give you real information if I am looking for a quote. I just want a quote, be thankful I even know who you are and that I am interested in your services.

In my opinion this application would fail usability testing. It's target audience is businesses. What do businesses want from your service?

• They want to know what it is you do
  
• They want to know how much it costs
  
• They want to be able to purchase the services when the time has come
  
• They don’t want to wait to find an answer out.

The last point holds true for anyone, anywhere. It’s not like I can’t just click somewhere else. Seriously, realise how far your competitor is away in Internet terms and then meet them.

So, how should the Facebook advertising web-app have been written?

If prices are fixed based on simple conditions, put them on the second page or have a link called “prices”. If the prices are based on a complex algorithm, then place all the controls on a single page and let the user tweak the setup to match their needs. It is not that hard.

My reasoning is so, firstly from a business perspective you are being open about your pricing policies. Honesty is good. Dishonesty equals zero business.

Secondly, as a user I want, when I want. I don’t have to jump through your hoops to achieve my goal. I don’t care about your hoops. I only care about my goals.

Thirdly, if you only display one price at the end of four pages of clicking one of three things will occur.

• The price is less than the budget for the advertiser. They may sign-up, but you may not be maximising your business potential. Secondly, they may have been willing to spend more but don’t know how to get those services from you.
  
• The price matches their budget. Not likely.
  
• The price is over their budget. Without a means to reduce their expectations to value they will go somewhere else.

If you display all the controls on one page then the following can occur:

• The user will tweak the control to what they wanted from you at a service level and the cost comes under their budget. This would have happened anyway using your existing setup.
  
• The user will tweak the controls to the absolute maximum of their budget. This is a win for both parties.
  
• The user will play with the figures and may end up spending just a little bit more if they can justify a potential benefit from it. This is a double win for you and a win for them.
  
• The user can’t find anything they like and they leave. However, they now know all about what you offer and what you cost. This is in their mind now and they won’t forget. There is a chance they will come back in the future.

By tailoring the application to the user’s needs rather than your own you do the following:

• You increase your chance of doing business.
  
• You appear to be a honest business entity.
  
• Your application
• Your application won’t piss people off and build a burning resentment within them.

Usability starts at the very beginning of software development and flows all the way through to User Acceptance Testing. You have to have in mind who you are developing the software for the end users before you write a line of code, before you write down a single requirement. For without a user, you have no need for an application.

Happy to discuss this delightful topic further, just drop me a line,

The sweetest thing

Every now and again I have a moment of realisation where I am thankful that somebody else thought of something before I did. I had one of these just the other day. They are a little bit like this xkcd comic.

I was trying to find a bug in a binary tree and the problem would not manifest itself immediately. The code worked for a seemingly arbitrary period of time, but at a depth of N a get out clause needed to be true and it wasn't. Therefore it kept on recursing until it either could not create any more branch-nodes or it generated a stack-overflow.

I attempted to step through my code trying to find my bug, but I find it hard to debug recursive code. You’re never too sure how deep you need to go before the problem manifests. After missing the bug I pondered how I could represent the tree to find the bug. One of the things I have been teaching myself to do of late, is when I have a hard to reproduce bug, ask myself the question "How can I best model my data to visualise the bug?". It is not quite changing my work practices to stop bugs before they occur, but it really helps once they do manifest.

XML came to my rescue. I added a dump routine that iterated the tree dumping it out in well formed xml. I then used Notepad++ to display the xml with indenting and the bug was immediately visible.

So the moral of the story is: I love xml and am glad somebody else thought of it a long time ago. It’s a sweet way to visualise certain types of data and with the assistance of text indentation it is even better.


Just so you don’t think I’m blinded by xml-lust: Xml for network messaging formats is still a bad idea. Seriously, any format where you spend 50% of your bandwidth describing the data being sent is wasteful.

p.s. I am aware that this is a bit of an odd post.

Please Sir, Can I have some more?

Facebook has a “More Ads” button. Who in their right mind would willingly request more advertising? Is this insanity of the highest order by the developers, or perhaps naivety?

QuickTest : Applying major changes to a repository

There are times when major changes need to be applied to a repository. Such an example is when a deployed application is moved from a local test server to a server in a pre-production environment. Nothing should change except for the URL, which unfortunately is hard-coded into every single object in your repository.

1. Open Object Repository Manager [Resources > Object Repository Manager]
2. Open your project repository [File > Open] Export to XML [File > Export to XML]
3. Enter a filename [exported.xml]
4. [ok]
5. Open you’re favourite XML editor (i.e. Notepad++)
6. Find and Replace on the current path to the new path
7. Do any other maintenance work
8. [Save xml-doc]
9. ensure document is well-formed
10. [Close xml-doc]
11. Meanwhile… Back in Object Repository Manager
    
12. Import from XML [File > Import from XML]
    
13. Save your Imported repository [imported.tsr]
    
14. In the file system, delete repository.tsr and exported.xml
    
15. Rename: imported.tsr to repository.tsr

Note: You will need to open a different QuickTest project and then reopen your current project for the results of the merge to be visible in the Object Repository. It’s a pain, but that’s the way of things.

QuickTest and CruiseControl for Continuous Integration

Originally I was working on the QuickTest CI setup with CruiseControl.NET (will work just as well for .java). This never got this completed for obvious reasons, but I thought I would share my notes with you.

Here is my original post detailing the qt2NUnit xsl and QuickTest structure

You need to launch QT remotely and I suspect this is more involved than what Grant did with TeamBuild. Look at what he did, and here is a reference to someone who executed QT remotely. I recommend using Grant's C# code to correct the repository references at run-time.

After that you just need a NAnt task to transform the results.xml doc. Here is a quicktest.build Nant task.

Then merge the results with your other result files. This works well if you have unit tests being produced in the same format. All the tests cases live together in harmony. This is a good thing.

Here is an excerpt of the publishers node within a CruiseControl build file.

You should now do some cleanup of temporary files, etc.


Note: Apologies for the images of xml. Blogger chucks a hissy fit if you include xml markup inside a post. What is this? The Nineties?

QuickTest and TeamBuild for Continuous Integration

A while ago Grant posted about integrating QuickTest (QT) into a Team Build continuous integration server. I said I would post about my half of the process. I’ve have been a bit busy of late to organise all my notes and publish this post. However it’s all organised now so, without further stalling for time…

The first part of the process is to ensure that you have a good QuickTest project setup This is what I did and why:

Organisation of Projects

An automated functional testing project (known to QuickTest as a Test) in my opinion should represent a single system. All test cases that actively test this system should be stored in this project. This ensures that our tests are all stored together.

A project should have a name and to simplify the integration with TeamBuild the project should be called “quick-test”. The folder that the quick-test project is in should be named after the Branch in source-control.

Example:
-The test team branch
|-The Name of the System
| |-quick-test

The common naming conventions will help ensure a close correlation between the code that is used to build a system and the automation code used to test it. This is especially relevant if the QT code is in a different branch to the app-code.

Note: Here there were two primary ways to configure the projects within TFS. The first is to put the functional testing code in the same branch as the code that it tests. The unit tests are also in that branch structure and with the code and all related content are together.

The second method is to have a separate branch for the Testing Team and then under each, a folder for each project. For convenience sake we named the sub-folder the same name as the code branch. Under this was our quick-test folder. Load-runner or any other tool that tests this same project would also be stored at this level.

We went with the second method because QT repository files can get quite large and this would have been a problem for the developers on an already strained TFS server. I also understand that workspace mappings may alleviate this problem further. I’m not 100% here as SVN is my source control system of choice (don’t tell Grant).

From an access management perspective, giving testers, who will probably be unfamiliar with the processes around source control, access to only one branch may make their lives simpler.

Even though we went with the second method, I would investigate the first method for your organisation to see if it fits. It’s a better logical structure.

Organisation of Repositories
The repository is where the objects that are used by QuickTest to control your application are stored. There are three levels of repositories:

• Local – Attached to an asset
• Project – All objects for a project
• Global – Merge of all project repositories

I recommend not using a local repository as it is a maintenance nightmare. As you develop your automation test cases, continually push your objects into the project level repository and associate all actions with the project one. It’s much easier to maintain that duplicating objects into each local repository.

Furthermore I don’t recommend using a global repository either. Your objects should relate to the project you are testing. Once again this is primarily a hassle to maintain. If you have dedicated individuals managing the merging of object repositories then go for it. Otherwise give it a miss.

Now, with the current setup, projects are very isolated and it is not possible to write automated scenarios that interact across separate systems. For this I would suggest having a common project that is basically the same as a global repository. The difference being that it only contains the objects needed for cross system test cases are included in this branch. This involves less merging maintenance than merging all objects.

If you require a common repository, put it into a separate branch. This will make it easier for the build script to pull in the common repository when running a continuous integration instance.

The repository should be called repository.tsr in all scenarios. The location of the repository will determine whether it is project, global or common repository. Once again this makes it easier for the build script as it can expect a particular file rather than the name of the file being explicitly configured.

Actions and Quick Test

My definition of actions in QuickTest differs from standard usage. Personally I don’t like their setup and put some rules around when actions should be used. This in my opinion makes their usage cleaner.

An action to QuickTest is a single script that can be executed in isolation. To use, an action will represent either a single test scenario or a reusable test scenario. Scripts can call other actions. However, for the sake of simplicity down the track a script should only call reusable actions.

I’ll define some terminology to make this consistent:

• Action – a single script that represents a single test case. Like all good test cases it should have single objective. If you are testing more than one thing, write two actions. See my post on the action-test-scenario naming convention for information on succinct naming of test cases and reasons for singular testing scope.
• Reusable Action – An action in QT that you can be flagged to be “reusable”. To me this should not generate and pass or fail counts. It should merely ensure a state when called.

The first action of a project should be called zzRepository and we can use that to edit and mange our project repository in a local space. This action is deliberately excluded from CI results.

Reusable actions should start with the word ensure. They are required to establish a state on behalf of the calling action (they ensure a state). Once again these actions are not included in the CI results.


Back to Continuous Integration

Once you have set up your structure. Talk with your TFS guy to get the code checked in an in the right format. Once again this link to Grant's work will cover the next step.

In summary: the build script will pull in the launch QT pulling in the repository.tsr. If you use a common repository, pull that in as well. Then pipe in all actions one at a time to execute them.

After the build occurs QT produces a results.xml document. We need to take this and make it look even slightly better than the default QT one to make it useful. I really like the way CruiseControl.NET presents the NUnit xml format. So I wrote a xsl transform to make the QT -> NUnit transform.

Qt2NUnit.xsl

Finally you will need to adjust the CruiseControl.NET xsl for presenting NUnit xml-docs to reference TeamBuild images. You can use my version here

It was funny then and is just as funny now

Many years ago I had a subscription to FHM. When I first subscribed I but my title as Sir. Once a month I would get a magazine delivered to Sir Chad Stone. I thought it was pretty funny.

FHM are currently going through a re-marketing phase to appear as more of a "thinking man's" magazine. A part of this campaign appears to be sending sample magazines to existing subscribers in an effort to get them to resubscribe.

In my letter box today was an FHM magazine address to Sir Chad Stone. It was even funnier today than it was back then because I had forgotten the punch line.

The meaning of life is... bug fixing?

News just in: life is too short for bug-fixing.

So if you're currently bug-fixing and you're fixing your own bugs. Have a think about how this bug came about. You should be able to come up with a way to ensure that the same kind of bug won't occur again or, at least, less frequently.

I can think of 107 things I would rather be doing than fixing bugs. I'm sure you can too.