Automated deployment - Why it is a good idea

Grant raised the concept of the automatic deployment in a comment on a post I made the other day about controlling test environments. I've been giving it a bit of thought of late and I've come up with a quick test you can do to see if you should automate your deployment process. As you go through the test you will also see various ways how automated deployment can improve your development practices.

Once Upon a Time...
Let me start with a tale of woe.

The past two weeks I've been working with a developer who has been lumped with some less than satisfactory code. Also lacking is a suitable development environment for him to work within. In an effort to get something suitable into test I have been working with him to solve the various problems. This past week has seen about15-25 deployments into test (don't get me started on unit testing). When he is not around another person from his team will do the work. Everyone of them at some point failed to deploy correctly.

Why? Firstly their application is made of a number of disparate components that all are deployed individually. They don't have version numbers to identify the latest build. They are deploying to a cluster. They are rushing to get late code into test. The testers don't have control of the test environment (yet).


Consider this
If your deployment is as simple as a point and click install and that is it, your failure points are: installing the right version and actually doing it (don't laugh). Two failure points, ignore the triviality for now. If you have a configuration file to manually adjust, there is another point. Add them up, one for each step you have to do in order. If you have deployment instructions. Add one more as those have to be followed, if you don't have deployment documentation add 1 million points. If you have to rebuild a fresh machine each time. Add 1 for each step. If you are smart and have a prepared image. Add a single point if you need to deploy it each time. Zero points if you don't.

I think you are starting to get what gives you points and what reduces your score. Stay with me as we go back to our example:

I don't know the full details of the application but I know there are about five components each of which needs to be deployed. So 5 * 3 (version, doing it, following instructions). Three of the installed components need to be turned on each time. So that is 3 more points. 18 is our score so far.

How many machines do you have to deploy to? One host? Score one for the number of targets Clustered array of four machines. Score 4. Pretty simple scoring system. Write this down as your targets score.

For our example we have a two hosts load balanced. So we score 2.

How frequently are you going to deploy your code? Once per iteration, how many iterations in the project? 10? Score one point each time you deploy per iteration, times iterations. Record this as frequency.

How many test environments are there? Dev, Dev-Int, Test, Non-functional, Release Candidate Integration, Pre-Production, Production? Here are seven from a in my opinion pretty regular configuration. 7 Points. Once again, in my example, just one, test. Add it up for your environment score.


Failure Points
Ok, so our formula for calculating the failure points of a deployment:

Failure Points = deployment-steps X frequency X environment X targets

Example = 18 x 2 x 15 = 540 points of failure

Not bad. You may argue that once you have tested your deployment that you shouldn't have any failures after that. It is tested after all. That is a good point, but remember we are not even talking about deployment testing here. Just vanilla dropping an application onto a box.

We (this is a team project after all, shared wins/shared losses) had 540 chances in a one week period to stuff up an aspect of the deployment process. Aside from the code failures, we had probably 10 deployment failures including not installing the code onto both machines in the cluster. Those particular defects are about as much fun to detect as a race condition.

Automated Deployment
How much you automate will directly impact the chances for deployment failure. Our two constants for the act of deployment were: actually doing it and installing the correct version.

Performing the work is now done by the auto-deployer. You still need to click the go button for certain environments. Automatic deployment implies that latest valid build is used so that problem is solved.

Individual deployment steps should be wrapped up into your installer. I mean every step. Installing software, opening ports on routers, configuration files. If you do some research you will find somebody has automated already it for you, or there is an API for it. If by chance that isn't done, do it yourself and then share the love.

Next up is the deployment to each machine on the cluster. Once again this should be handled in by your autodeployer. So that one is fixed, score a zero.

After that was the total number of deployments. That shouldn't change. As long as your autodeployer is operational and you click the go button as required. You should be down to a score of 5 (once for each environment from test afterwards).

With our example we should go from 540 failure points to 5. One for each deployment that has occured over the past week. Triggered by the test team as required. There are no other manual steps.

Bonus Feature
If the latest build is unusable for testing. Allow the testers to flag it as so (Build Quality) and have the autodeployer ignore that build for future deployments.


Conclusion
You may realise by now, that I have been a little bit over the top with my example. Furthermore, every iteration you don't deploy to every environment. You and I know this, but it won't change your score that much. You may also think of more places in which the scoring system should change. Post them as a comment and I'll put together a little spreadsheet you can use.

I am not going to tell you how to automate your deployment process. I've got an idea on one way to do it and I'll post about when I've done it. In the meantime here are a couple of other ideas to get you started (thanks to Grant for these):

• Use PsExec
  
• Use putty if you are not on a windows box
  
• Via TFS Build here and here
  

Before I go some more juicy content: your autodeployer should not be used until you have tested it through all environments. Including deployment into production.

Controlling Testing Environments

Why You Should Care?
Testing environments are fundamental to successful testing. The test environment is where testing occurs and without a controlled, regulated, stable testing environment you are undermining your entire testing foundation. Scary stuff!

What do I mean by controlling a testing environment? I mean ensuring:

• that you know that each environment has the correct code,
• that the various integrating applications have compatible versions,
• that the correct hardware and software configuration exists,
• that the data is legitimate and in the right quantities,
• access to the environment is restricted and,
• security policies mimic production

All of above items combine to make a stable, controlled, test environment.

Without proper management of testing environments whenever a defect is identified you have to:

1. identify the software build,
2. determine how long that build has been there,
3. determine if there is a later build available?
4. ensure that the date is valid?
5. review the hardware to ensure it matches production
6. review the additional software components to ensure it matches production

Beyond environmental stability there are particular test scenarios that you can now perform. You can engage in deployment testing. Every release the software package is released into production. How often is this software deployment process tested?

Other benefits are: when you receive a "bad" build you can un-install it and re-install the previous one until it gets fixed. Or, you can get two competing builds from the development team and compare them for performance. I am doing this one next week.


So how do we go about doing this?
The first step is to identify how many test environments you have / need. In summary, I like to see at least the following:

• Development - one per developer, usually the development box but ideally should be a VM or similar that matches production architecture/operating system/software configuration. Developers may call it a build box, but they do unit testing here, so it is a test environment.
• Development integration - one per project/release. Here the development team works on integrating their individual components together.
• Test - where the brunt of the tester's work is done. There should be a dedicated environment for each project.

The following environments can usually be shared between project teams depending on the number and types of projects being developed concurrently.

• User acceptance testing - can be done in other environments if the resources are not available. Ideally should be a dedicated environment that looks like prod + all code between now and project release. This is an optional environment in my opinion as there are lots of good places to do UAT and it really depends on the maturity of your project and your organisation's available infrastructure.
• Non-functional - performance, stress, load, robustness - should be identical infrastructure to production, the data requirements can exceed production quantities but must match it in authenticity.

More environments are possible. I didn't cover integration or release candidate environments (you may have duplicate environments or subsets for prod-1, prod and prod+1) and it really depends on the number of software products being developed concurrently. I won't be discussing the logistics of establishing test environments here nor how to acquire them cheaply.

To actually gain control. First talk to the development team about your requirements for a stable testing environment. Explain your reasons and get their support. The next step is not always necessary but can give you good piece of mind. Remove developer access to the test environments. I am talking about everywhere, web servers, databases, terminal services, virtual machines. If its apart of the testing environment they should stay out.

It isn't because you don't trust them. After deployment you probably shouldn't be on those machines either. Sure, there are some testing scenarios where getting into the nitty gritty is required, but not always and certainly not when testing from the user's perspective. The bottom line is that the less people who have access to these machines results in a smaller chance of accidental environmental comprise.


So what aspects do we control?
Primarily we need to control the Entry and Exit criteria to each environment. The first step is the development environment. Entry is entirely up to the developer and exit should be achieved when unit tests passed. As the next step is the development integration environment, the development lead should control code entry.

Entry into the test environment: regardless of the development methodology the delivery to test should be scheduled. Development completes a build that delivers "N chunks" of functionality. Unit tests have passed and they are good to go.

Developers should then prepare a deployment package (like they will for the eventual production release) and place it in a shared location that the deployment testers can access. It is now up to the deployment testers to deploy the code at the request of the project testing team (these are quite often the same team). Once a build has been deployed, some build verification tests are executed (preferably automated) and the testers can continue their work.

To move from test into any environment afterwards (release candidate integration, pre-production, etc) depends on the organisation but usually the following: Testing has been completed, defects resolved, user documentation produced and most importantly user sign-off has been acquired.

The final environments (pre-production, etc) are usually (should be) managed by a release manager who controls the entry and exit gates from each environment after test and on into production. I won't cover these here.


Evidence or it never happened!
Example A: About a month ago we had a problem where one of our test environments wasn't working as expected. It took the developer over a week to find the problem. Turns out another developer had promoted some code without letting anyone else know. The code didn't work and he left it there.

This could have been avoided if the developer didn't have access to deploy in our environment. Unfortunately he does, but it is something that we are working towards rectifying.

Example B: I once worked on a project that had five development teams. Two database groups and three code cutters. Had they been able to deploy when they wanted, our test environment would have been useless. None of the teams were ever ready at the same time and it would have meant we would have had code without appropriate database support. Components that were meant to integrate but did not match because the latest build of application x wasn't ready yet.

By waiting until all builds were ready and running through the deployment ourselves we ensured that our test environment was stable and had the same level of development progression all the way through.


Too much information, summarise before I stop caring!

1. Controlling Test Environments = Good
2. Focus on developing entry and exit criteria
3. Build up to production-like environments - each successive environment should be closer and closer to production.
4. Evolve towards the goal of environmental control rather than a big bang approach. Some transitions will take longer than others (i.e. getting the right hardware) so pick a level of control for each release, get everyone involved and implement it.
5. Get team buy in (developers, testers) - education is the key
6. Don't make the entry into the test environment documentation heavy.

It all looks too easy, how could this go wrong?
Get development buy-in. This is important you don't want to alienate the development team. Not all developers or development teams are inconsiderate, nor do they have ulterior motives. Usually it's a simple lack of awareness and discussing with them the direction you want to take with the testing environments will achieve two things. Firstly, they have greater visibility into the testing arena and secondly they often realise that they can help improve quality by doing less. Who doesn't like doing that?


Don't make it complicated: The goal of this is to achieve a high quality test environment to facilitate high quality testing. Don't produce a set of forms and a series of hoops that you need to force various developers and teams to fill out whilst jumping through. They won't like it and they probably won't like you.

When I first tried locking down an environment, I asked the developers to fill out a handover to test document that listed the build, implemented task items, resolved defects and similar items. I had buy in and for the first few cycles it worked ok. It wasn't great though. All I was doing was accumulating bits of paper and wasting their time by making them fill it out.

All I do these days is discuss with the developers the reasons why the environment needs to be locked down and to let me know when a new build is ready. I'm usually involved in iteration planning meetings so I know what is coming anyway. All that waffle they had to fill out is automatically generated from defect management, task management and source control software.

My testing environments are generally stable, developers are happy to hand me deployment packages and consider deployment defects just as important as normal defects. After all, deployment is the first chance a piece of software has to fail in production. It is also the first place user's will see your application.

It takes time to move towards a controlled environment and as you read in my examples, my employer is not there yet either, but we are getting closer.


One other note: You may not have the ability (whether technical or organisational) to perform development testing. See if you can organise to sit with the technical team that does deployments for you.