Once Upon a Time...
Let me start with a tale of woe.
The past two weeks I've been working with a developer who has been lumped with some less than satisfactory code. Also lacking is a suitable development environment for him to work within. In an effort to get something suitable into test I have been working with him to solve the various problems. This past week has seen about15-25 deployments into test (don't get me started on unit testing). When he is not around another person from his team will do the work. Everyone of them at some point failed to deploy correctly.
Why? Firstly their application is made of a number of disparate components that all are deployed individually. They don't have version numbers to identify the latest build. They are deploying to a cluster. They are rushing to get late code into test. The testers don't have control of the test environment (yet).
Consider this
If your deployment is as simple as a point and click install and that is it, your failure points are: installing the right version and actually doing it (don't laugh). Two failure points, ignore the triviality for now. If you have a configuration file to manually adjust, there is another point. Add them up, one for each step you have to do in order. If you have deployment instructions. Add one more as those have to be followed, if you don't have deployment documentation add 1 million points. If you have to rebuild a fresh machine each time. Add 1 for each step. If you are smart and have a prepared image. Add a single point if you need to deploy it each time. Zero points if you don't.
I think you are starting to get what gives you points and what reduces your score. Stay with me as we go back to our example:
I don't know the full details of the application but I know there are about five components each of which needs to be deployed. So 5 * 3 (version, doing it, following instructions). Three of the installed components need to be turned on each time. So that is 3 more points. 18 is our score so far.
How many machines do you have to deploy to? One host? Score one for the number of targets Clustered array of four machines. Score 4. Pretty simple scoring system. Write this down as your targets score.
For our example we have a two hosts load balanced. So we score 2.
How frequently are you going to deploy your code? Once per iteration, how many iterations in the project? 10? Score one point each time you deploy per iteration, times iterations. Record this as frequency.
How many test environments are there? Dev, Dev-Int, Test, Non-functional, Release Candidate Integration, Pre-Production, Production? Here are seven from a in my opinion pretty regular configuration. 7 Points. Once again, in my example, just one, test. Add it up for your environment score.
Failure Points
Ok, so our formula for calculating the failure points of a deployment:
Failure Points = deployment-steps X frequency X environment X targets
Example = 18 x 2 x 15 = 540 points of failure
Not bad. You may argue that once you have tested your deployment that you shouldn't have any failures after that. It is tested after all. That is a good point, but remember we are not even talking about deployment testing here. Just vanilla dropping an application onto a box.
We (this is a team project after all, shared wins/shared losses) had 540 chances in a one week period to stuff up an aspect of the deployment process. Aside from the code failures, we had probably 10 deployment failures including not installing the code onto both machines in the cluster. Those particular defects are about as much fun to detect as a race condition.
Automated Deployment
How much you automate will directly impact the chances for deployment failure. Our two constants for the act of deployment were: actually doing it and installing the correct version.
Performing the work is now done by the auto-deployer. You still need to click the go button for certain environments. Automatic deployment implies that latest valid build is used so that problem is solved.
Individual deployment steps should be wrapped up into your installer. I mean every step. Installing software, opening ports on routers, configuration files. If you do some research you will find somebody has automated already it for you, or there is an API for it. If by chance that isn't done, do it yourself and then share the love.
Next up is the deployment to each machine on the cluster. Once again this should be handled in by your autodeployer. So that one is fixed, score a zero.
After that was the total number of deployments. That shouldn't change. As long as your autodeployer is operational and you click the go button as required. You should be down to a score of 5 (once for each environment from test afterwards).
With our example we should go from 540 failure points to 5. One for each deployment that has occured over the past week. Triggered by the test team as required. There are no other manual steps.
Bonus Feature
If the latest build is unusable for testing. Allow the testers to flag it as so (Build Quality) and have the autodeployer ignore that build for future deployments.
Conclusion
You may realise by now, that I have been a little bit over the top with my example. Furthermore, every iteration you don't deploy to every environment. You and I know this, but it won't change your score that much. You may also think of more places in which the scoring system should change. Post them as a comment and I'll put together a little spreadsheet you can use.
I am not going to tell you how to automate your deployment process. I've got an idea on one way to do it and I'll post about when I've done it. In the meantime here are a couple of other ideas to get you started (thanks to Grant for these):
Before I go some more juicy content: your autodeployer should not be used until you have tested it through all environments. Including deployment into production.
No comments:
Post a Comment