Controlling Testing Environments

Why You Should Care?
Testing environments are fundamental to successful testing. The test environment is where testing occurs and without a controlled, regulated, stable testing environment you are undermining your entire testing foundation. Scary stuff!

What do I mean by controlling a testing environment? I mean ensuring:

• that you know that each environment has the correct code,
• that the various integrating applications have compatible versions,
• that the correct hardware and software configuration exists,
• that the data is legitimate and in the right quantities,
• access to the environment is restricted and,
• security policies mimic production

All of above items combine to make a stable, controlled, test environment.

Without proper management of testing environments whenever a defect is identified you have to:

1. identify the software build,
2. determine how long that build has been there,
3. determine if there is a later build available?
4. ensure that the date is valid?
5. review the hardware to ensure it matches production
6. review the additional software components to ensure it matches production

Beyond environmental stability there are particular test scenarios that you can now perform. You can engage in deployment testing. Every release the software package is released into production. How often is this software deployment process tested?

Other benefits are: when you receive a "bad" build you can un-install it and re-install the previous one until it gets fixed. Or, you can get two competing builds from the development team and compare them for performance. I am doing this one next week.


So how do we go about doing this?
The first step is to identify how many test environments you have / need. In summary, I like to see at least the following:

• Development - one per developer, usually the development box but ideally should be a VM or similar that matches production architecture/operating system/software configuration. Developers may call it a build box, but they do unit testing here, so it is a test environment.
• Development integration - one per project/release. Here the development team works on integrating their individual components together.
• Test - where the brunt of the tester's work is done. There should be a dedicated environment for each project.

The following environments can usually be shared between project teams depending on the number and types of projects being developed concurrently.

• User acceptance testing - can be done in other environments if the resources are not available. Ideally should be a dedicated environment that looks like prod + all code between now and project release. This is an optional environment in my opinion as there are lots of good places to do UAT and it really depends on the maturity of your project and your organisation's available infrastructure.
• Non-functional - performance, stress, load, robustness - should be identical infrastructure to production, the data requirements can exceed production quantities but must match it in authenticity.

More environments are possible. I didn't cover integration or release candidate environments (you may have duplicate environments or subsets for prod-1, prod and prod+1) and it really depends on the number of software products being developed concurrently. I won't be discussing the logistics of establishing test environments here nor how to acquire them cheaply.

To actually gain control. First talk to the development team about your requirements for a stable testing environment. Explain your reasons and get their support. The next step is not always necessary but can give you good piece of mind. Remove developer access to the test environments. I am talking about everywhere, web servers, databases, terminal services, virtual machines. If its apart of the testing environment they should stay out.

It isn't because you don't trust them. After deployment you probably shouldn't be on those machines either. Sure, there are some testing scenarios where getting into the nitty gritty is required, but not always and certainly not when testing from the user's perspective. The bottom line is that the less people who have access to these machines results in a smaller chance of accidental environmental comprise.


So what aspects do we control?
Primarily we need to control the Entry and Exit criteria to each environment. The first step is the development environment. Entry is entirely up to the developer and exit should be achieved when unit tests passed. As the next step is the development integration environment, the development lead should control code entry.

Entry into the test environment: regardless of the development methodology the delivery to test should be scheduled. Development completes a build that delivers "N chunks" of functionality. Unit tests have passed and they are good to go.

Developers should then prepare a deployment package (like they will for the eventual production release) and place it in a shared location that the deployment testers can access. It is now up to the deployment testers to deploy the code at the request of the project testing team (these are quite often the same team). Once a build has been deployed, some build verification tests are executed (preferably automated) and the testers can continue their work.

To move from test into any environment afterwards (release candidate integration, pre-production, etc) depends on the organisation but usually the following: Testing has been completed, defects resolved, user documentation produced and most importantly user sign-off has been acquired.

The final environments (pre-production, etc) are usually (should be) managed by a release manager who controls the entry and exit gates from each environment after test and on into production. I won't cover these here.


Evidence or it never happened!
Example A: About a month ago we had a problem where one of our test environments wasn't working as expected. It took the developer over a week to find the problem. Turns out another developer had promoted some code without letting anyone else know. The code didn't work and he left it there.

This could have been avoided if the developer didn't have access to deploy in our environment. Unfortunately he does, but it is something that we are working towards rectifying.

Example B: I once worked on a project that had five development teams. Two database groups and three code cutters. Had they been able to deploy when they wanted, our test environment would have been useless. None of the teams were ever ready at the same time and it would have meant we would have had code without appropriate database support. Components that were meant to integrate but did not match because the latest build of application x wasn't ready yet.

By waiting until all builds were ready and running through the deployment ourselves we ensured that our test environment was stable and had the same level of development progression all the way through.


Too much information, summarise before I stop caring!

1. Controlling Test Environments = Good
2. Focus on developing entry and exit criteria
3. Build up to production-like environments - each successive environment should be closer and closer to production.
4. Evolve towards the goal of environmental control rather than a big bang approach. Some transitions will take longer than others (i.e. getting the right hardware) so pick a level of control for each release, get everyone involved and implement it.
5. Get team buy in (developers, testers) - education is the key
6. Don't make the entry into the test environment documentation heavy.

It all looks too easy, how could this go wrong?
Get development buy-in. This is important you don't want to alienate the development team. Not all developers or development teams are inconsiderate, nor do they have ulterior motives. Usually it's a simple lack of awareness and discussing with them the direction you want to take with the testing environments will achieve two things. Firstly, they have greater visibility into the testing arena and secondly they often realise that they can help improve quality by doing less. Who doesn't like doing that?


Don't make it complicated: The goal of this is to achieve a high quality test environment to facilitate high quality testing. Don't produce a set of forms and a series of hoops that you need to force various developers and teams to fill out whilst jumping through. They won't like it and they probably won't like you.

When I first tried locking down an environment, I asked the developers to fill out a handover to test document that listed the build, implemented task items, resolved defects and similar items. I had buy in and for the first few cycles it worked ok. It wasn't great though. All I was doing was accumulating bits of paper and wasting their time by making them fill it out.

All I do these days is discuss with the developers the reasons why the environment needs to be locked down and to let me know when a new build is ready. I'm usually involved in iteration planning meetings so I know what is coming anyway. All that waffle they had to fill out is automatically generated from defect management, task management and source control software.

My testing environments are generally stable, developers are happy to hand me deployment packages and consider deployment defects just as important as normal defects. After all, deployment is the first chance a piece of software has to fail in production. It is also the first place user's will see your application.

It takes time to move towards a controlled environment and as you read in my examples, my employer is not there yet either, but we are getting closer.


One other note: You may not have the ability (whether technical or organisational) to perform development testing. See if you can organise to sit with the technical team that does deployments for you.