Automated deployment - Why it is a good idea

Grant raised the concept of the automatic deployment in a comment on a post I made the other day about controlling test environments. I've been giving it a bit of thought of late and I've come up with a quick test you can do to see if you should automate your deployment process. As you go through the test you will also see various ways how automated deployment can improve your development practices.

Once Upon a Time...
Let me start with a tale of woe.

The past two weeks I've been working with a developer who has been lumped with some less than satisfactory code. Also lacking is a suitable development environment for him to work within. In an effort to get something suitable into test I have been working with him to solve the various problems. This past week has seen about15-25 deployments into test (don't get me started on unit testing). When he is not around another person from his team will do the work. Everyone of them at some point failed to deploy correctly.

Why? Firstly their application is made of a number of disparate components that all are deployed individually. They don't have version numbers to identify the latest build. They are deploying to a cluster. They are rushing to get late code into test. The testers don't have control of the test environment (yet).


Consider this
If your deployment is as simple as a point and click install and that is it, your failure points are: installing the right version and actually doing it (don't laugh). Two failure points, ignore the triviality for now. If you have a configuration file to manually adjust, there is another point. Add them up, one for each step you have to do in order. If you have deployment instructions. Add one more as those have to be followed, if you don't have deployment documentation add 1 million points. If you have to rebuild a fresh machine each time. Add 1 for each step. If you are smart and have a prepared image. Add a single point if you need to deploy it each time. Zero points if you don't.

I think you are starting to get what gives you points and what reduces your score. Stay with me as we go back to our example:

I don't know the full details of the application but I know there are about five components each of which needs to be deployed. So 5 * 3 (version, doing it, following instructions). Three of the installed components need to be turned on each time. So that is 3 more points. 18 is our score so far.

How many machines do you have to deploy to? One host? Score one for the number of targets Clustered array of four machines. Score 4. Pretty simple scoring system. Write this down as your targets score.

For our example we have a two hosts load balanced. So we score 2.

How frequently are you going to deploy your code? Once per iteration, how many iterations in the project? 10? Score one point each time you deploy per iteration, times iterations. Record this as frequency.

How many test environments are there? Dev, Dev-Int, Test, Non-functional, Release Candidate Integration, Pre-Production, Production? Here are seven from a in my opinion pretty regular configuration. 7 Points. Once again, in my example, just one, test. Add it up for your environment score.


Failure Points
Ok, so our formula for calculating the failure points of a deployment:

Failure Points = deployment-steps X frequency X environment X targets

Example = 18 x 2 x 15 = 540 points of failure

Not bad. You may argue that once you have tested your deployment that you shouldn't have any failures after that. It is tested after all. That is a good point, but remember we are not even talking about deployment testing here. Just vanilla dropping an application onto a box.

We (this is a team project after all, shared wins/shared losses) had 540 chances in a one week period to stuff up an aspect of the deployment process. Aside from the code failures, we had probably 10 deployment failures including not installing the code onto both machines in the cluster. Those particular defects are about as much fun to detect as a race condition.

Automated Deployment
How much you automate will directly impact the chances for deployment failure. Our two constants for the act of deployment were: actually doing it and installing the correct version.

Performing the work is now done by the auto-deployer. You still need to click the go button for certain environments. Automatic deployment implies that latest valid build is used so that problem is solved.

Individual deployment steps should be wrapped up into your installer. I mean every step. Installing software, opening ports on routers, configuration files. If you do some research you will find somebody has automated already it for you, or there is an API for it. If by chance that isn't done, do it yourself and then share the love.

Next up is the deployment to each machine on the cluster. Once again this should be handled in by your autodeployer. So that one is fixed, score a zero.

After that was the total number of deployments. That shouldn't change. As long as your autodeployer is operational and you click the go button as required. You should be down to a score of 5 (once for each environment from test afterwards).

With our example we should go from 540 failure points to 5. One for each deployment that has occured over the past week. Triggered by the test team as required. There are no other manual steps.

Bonus Feature
If the latest build is unusable for testing. Allow the testers to flag it as so (Build Quality) and have the autodeployer ignore that build for future deployments.


Conclusion
You may realise by now, that I have been a little bit over the top with my example. Furthermore, every iteration you don't deploy to every environment. You and I know this, but it won't change your score that much. You may also think of more places in which the scoring system should change. Post them as a comment and I'll put together a little spreadsheet you can use.

I am not going to tell you how to automate your deployment process. I've got an idea on one way to do it and I'll post about when I've done it. In the meantime here are a couple of other ideas to get you started (thanks to Grant for these):

• Use PsExec
  
• Use putty if you are not on a windows box
  
• Via TFS Build here and here
  

Before I go some more juicy content: your autodeployer should not be used until you have tested it through all environments. Including deployment into production.

Bloggerscript

Ok, I think the javascript in Blogger is a little inefficient. As I type into this dialog box my CPU usage cranks up to about 80-90%. According to procexp it's Firefox, which means Blogger. I'm not entirely sure what is going on in each keystroke but this is ridiculous. I'm waiting for each letter to appear on the screen. I'm not waiting long but it's making writing a post difficult.

Having a look at the source and there isn't much occurring. The "Save Now" being enabled on keydown but not much else. It could be a Firefox's Javascript processor. Firefox did update before I started the session but I wouldn't have thought it was that.

I am not working on my primary machine though, which maybe why I've never noticed this before. I am working on a machine that is just used for Continuous Integration and serving up SVN. So it's not powerful at all. I'll explain in my next post while I'm not using my primary machine.

This machine: 1.7Ghz Intel i386 with 1GB of RAM (DDR type1 probably)
Primary machine: Dual Core AMD 4200 64bit (32bit OS) with 2GB of 800mhz DDR2 RAM

Personally I think web-applications should be transparent and I realise I'm not the only one who thinks so. I'm hardly covering new ground. Web-applications are hard to develop properly (super-easy to do poorly... I know, I've written a few), you have a bunch of major platforms you need to support and you don't have the luxury of building a binary/distribution package to suite each one. I built www.icnh-games.com myself (well not entirely, I had a separate company do the style-sheet and layout, non-programmer-art, ftw) and I wrote Perl scripts to dynamically build the content from in-game xml-documents. However, it took us about two months after it was completed to iron out all the cross platform bugs (and I'm sure there are still some in there hiding... a testers job is rarely allowed to be done).

However, right from the start we stipulated with the company we contracted [Voodoo] that we needed to support the primary browsers (IE6, IE7, Firefox, Safari, Opera and there was another... don't feel offended if I left you out) as well as handling users that had Flash (inline vids) didn't have Flash (static images) and may or may not have Javascript (to support fancy menus as well as automatically starting the Flash movies rather than waiting for the user). Not to mention that multiple Flash versions that are out there. Still we got it all working in harmony. The more "features" you have enabled the better your experience is. The important thing is not to deny an experience simply because one is too lazy to support a user's application or operating system choice. We are here to provide applications for users and denying a user simply because of their operating system or browser choice seems a little bit too close to [racial|sexual|religious|*]-discrimination for my liking. Virtual discrimination is what it should be called. It makes it sounds like a social taboo and certainly no where near as cool as supporting Linux only cause everyone else is a n00b, or Apple or Microsoft. Also, it's not as funny as Apartheight.

I will post a blog (in a month or so once I've got all the bugs ironed out and can take comparative screenshots) that illustrates the multiple paths through the ICNH Games Framework. We support Windows (XP, Vista), Apple (10.4, 10.5 and I'm doing some checks as we speak to find out if we can support 10.3 as well... I knows it's old but I have a 10.3 box right here) and Linux (Ubuntu, FreeBSD, Fedora, OpenSUSE, MEPIS and PC Linux OS, potentially more but not tested). On top of that we provide distributions for i386, x86_64 (Intel or AMD), PowerPC 32 and 64 bit as well as there hardware features combinations of MMX, SSE, SSE2, SSE3.1 and SSE4 and 3DNOW. At the GPU level we have paths through our rendering engine that range from no-shaders, no-multitexture and no-VBOs to full-shader support, high dynamic range rendering with countless texture slots (my GPU supports 128 texture units apparently... I've never tested it) and batching algorithms to squeeze as much performance out of the machine as I know how to. I won't say as much as possible, as I know I'm not great at assembly, I'm only good at designing for performance and not getting down into the assembly level code and tweaking it further. One day a clever dev will come in and make it faster than it was before and I'll be happier.

All this being said the visual experience provided to low-end users (like my laptop with the awesome integrated graphics card) is not going to be mind-blowingly awesome, but for No Horizons and potentially other games, users will still be able to play the game. I'm will always play a game I like that looks bad over a game that looks awesome and is just a bit rubbish. That is pretty much our number point at ICNH Games: How many people can we get to play this game? Equal with it is: Is the game fun?

This was only possible because at the start we wrote down that we wanted to support all users, not just those running triple SLI. If you plan to support multiple environment configurations then you are going to get a lot closer to supporting them all then you are by tacking the support on at the end. Refactoring architectural solutions to support user XYZ is only going to make you resent them.

I'm not sure if Google's Blogger is designed to run on all browser configurations. I'm sure they have tested it, but have they tested it running a crappy old PC box? Moore's law has the speed of computers doubling every 18 months. This just means that the percentage of crappy computers is steadily increasing.



Before I go, I don't want to turn this into a gripe-blog but a few things have been bothering me of late and I never sleep much so that can't help. Is it that they've always bothered me and that starting my blog is just a way to vent existing frustrations or are all these recent occurrences of discontent purely anomalies in a generally happy and thoroughly enjoyable life?


This could be an interesting research topic; "Does starting a blog cause one to complain more frequently (or just more openly) than what they did before?". I'm sure than anonymity of the interwub certainly helps here. Perhaps it's the fact that blogging can be very cynical (not all of it is, but I've sure read a few cynical blogs in my time) and therefore bitching on a blog is more acceptable than bitching in person. It certainly allows one the escape of the social stigma associated with their gender. For instance the act of complaining in a male-oriented "traditional" Australian society is frequently met with responses like "dry your eyes Princess" and "harden the fuck up". This isn't all bad of course (fyi, nothing is ever all-bad or all-good) as Australian's tend to be good at putting up with the crap and getting on with the hard work.

I won't get into a discussion on Australian culture (which for the most part I love). For those who have already done the research or are doing it. Drop me a line, I find research topics fascinating.

Project Development Iterations

One of the things I like about where I work is the freedom different development teams have in trying new things with respect to work practices. We are currently starting version 1.5 of a project and one of the developers wanted to try a different way of structuring our iterations to minimise the issues we had in version 1.0 and to improve the overall quality of the project being developed.

The main issue we wanted to fix is developers getting too far ahead of testers - we had a two week iterative process where the developers would code for two weeks straight and then on day 10 they would provide a build to test. Then they would continue on as fast as they could.

This was a problem, they kept on getting further and further away from the testers because new code ranked higher in their eyes than old code did. Therefore some bug fixes took a long time to come, this hampered testing.

So one of the developers suggested a different iteration configuration. An iteration still lasts two weeks but there are five days for development of new code and five days for testing. The final day of development involves preparing the deployment for test. The final day of testing involves preparing the deployment for User Acceptance Testing.


Today was the first day of the first testing phase and so far so good. Defects were raised (not that many which impressed me for day one code) and fixes are already being prepared. Beyond that developers have realised they've made a mistake in one place and as they have the time, they are checking for similar problems in other areas. Proactive bug fixing, another plus.


Now, you may be wondering how complex code can be with five days of development including unit testing. Well for starters the number of developers double the testers but this is a small iteration task wise. Not all will be though so we are breaking our iteration tasks up into two groups. Single iteration tasks and double iteration tasks.

Every second iteration will see the deployment of more complex functionality that cannot be implemented in 5 days (effectively 15 days). As testers I am happy with this, we know these larger tasks are coming in advance and can plan for their arrival. We also still get five days of dedicated developer bug fixing after its delivery.

Both the development and testing teams are aware that the potential for the number of defects to still be greater than what the developers can fix in five days (especially if there are some doozies). To combat this (as what should have always be done), defects have priority one in the iteration following their discovery and as testers we're obliged to ensure that all new code gets at least a once-over to gauge the quality of the build.

Time will tell if policy this works, but at least we are working together to solve our past mistakes.

Default Parameters

As of today, I no longer like default parameters. Three times in the past week I have been caught out by default parameters, some of it in my own code! You would think I would know what is going on.

I like default parameters, I think they are are good way of allowing optional parameters on interfaces and they can provide an automatic suite of function overloads. However, when it comes to refactoring and debugging they can be hazardous.

The first scenario was when I was refactoring some collection code to accept a "catalogue" type that would define what is allowed in the collection. This code wasn't my own but I had designed the overall solution. Anyway every parameter on the collection constructor had a default value (one may argue, why have parameters at all, but there was a fair bit going on behind the scenes). I removed some no redundant parameters and added my catalogue. No compiler errors but the code didn't work. I had made changes to about 3 classes for this change, so it took a little while to work out what I had done wrong.

I had forgotten to take a parameter out of some calling code, and because the defaults were optional, this parameter was pushed down the list. Hence no compiler error, and technically a valid object, it just wasn't what my code was expecting. A while later we're in Crash-ville. Naturally I should have (and this was how I resolved it) found the references to the constructor and made sure they were correct. This was only a right-click away but I had gotten lazy on compiler warnings.

The second scenario was a bit of a doozy. It relates to the same set of objects. The catalogue has a function that when an object is passed to it, it returns true if the object is a match for the catalogue, false if it isn't. There is a default parameter that specifies whether the object passed in should update some information inside the catalogue. I only needed the update in a few places (less than I don't need it), so the default is false (or don't update).

my code reads something like this:

if (collection->getCatalogue()->isMatch
                 ((*objectItr)->m_element))
{
 // we have our match, catalogue is unchanged
}

//OR

if (collection->getCatalogue()->isMatch
                 ((*objectItr)->m_element), true)
{
 // we have a match and our catalogue was updated
}

Apologies for the formatting. I need to resize this blog, it's like trying to write on the side of a milk carton. Just imagine the if-statement all on one line.

Now, that code compiles and works fine for a number of test scenarios. However, it always evaluates to 'true'. The reason is, the true at the end of the if-statement should be a parameter inside isMatch, but it isn't so isMatch defaults (giving me incorrect logic) and the true is evaluated by the comma-operator. Fun-times.

The correct if-statement is:

if (collection->getCatalogue()->isMatch
                 ((*objectItr)->m_element, true))
{
 // we have a match and our catalogue was updated
}

With the context removed my original code is resembling something similar to this:

if (condition, true)
{
 //always evaluate
}

In reality I should have named my methods better (Yes, I know, I harp on about test-case naming conventions and then do this).

• isMatch (object)
  
• isMatchWithCatalogueUpdate (object)

or potentially

• isMatch(object)
  
• UpdateCatalogue (object) --called within the if-statement

I no longer have default parameters and I do have code that works. I suspect my usage of default parameters will approach zero from here on in. I won't discount them entirely as that is foolish, I'll just think twice before using them.

Testing SOA - an introduction

I might as well start with a fairly large topic and one that puts me in, I feel, an interesting position. The testing of Web services is inherently a technical task in the realm of generally non-technical testers.

There are a number of products out there that attempt to simplify the process of testing services for testing personnel. HP's Service Test, Parasoft's SOAtest, soupUI, etc, but in reality they are just UI wrappers that generate code facilitating communication with the service. Sure they may it easier than cutting your own code to do the same thing and I'm sure some of them auto-generate boundary tests (I don't know if they do but it's not that hard or unreasonable so I'll assume it) but deep down there are couple of issues nagging at me.

Should testers be testing services in the first place?

I've heard some good arguments for both and I've yet to make up my mind. This is really the basis for all my questions, can I justify who is going to test service code? My decision will impact the direction our organisation takes, so this question, I take very seriously. Currently I see five paths:

• Developers test all service code (unit, functional, performance, etc, etc)
• Developers perform unit and functionality testing and testers perform performance, scalability, availability, etc.
• Testers produce a test-specification which they hand to the developer who will code the tests
• Testers cut the code themselves (requires a technical-tester)
• Testers use a service testing tool

How should the testing of services be performed?

Using an SOA test tool? Cutting code? I'll get onto this in more detail once I've given the various tools a solid workout but they haven't convinced me yet and while that isn't a complete write off, I feel if you can't prove your worth immediately, in the one task you do, you're not effective. Then again, cutting code is prone to human error and having to write repetitive code to test a service, arduous. Code Generation anyone?

How will the testing environment be structured?

Testing services from any perspective is hardly trivial. There are deployment and configuration issues and these impact the viability of testing. I know that it is very easy to spawn a thread at the start of a MS-Test object that creates a service. It then notifies the main test-thread it is ready and testing can commence. The unit-test cases are executed and then everything is cleaned up. This setup would allow a developer to prove a service in the comfort of his own box. It also allows the test cases to be placed on the end of a continuous integration run. However, that deployment of the service will not match how the service is deployed in production. Therefore our testing is only good enough to prove functionality. This leads me to the next question:

Who, and how are we going to test services for Performance, Scalability, Robustness, Load/Stress, Availability, Configuration and Deployment?

These testing scenarios are complex and often time consuming. Will this move into the realm of the developer or will they remain world of testers? I feel that testing most of the above scenarios requires prod-like configuration and deployment. This automatically moves them out of the development environment where stability is mandated.

What defines a service from a documentation perspective?

Services are an IT solution to a Business problem and because of that a requirement specification is not going to detail the interfaces on a service. That belongs in a technical specification. Enter Agile and my chances of seeing such specifications are reduced (apologies to all those that use and like Agile. I like it too but in a world where specifications are thin on the ground Agile doesn't appear to make it any easier).

How do you determine if a service is fit for production?

This may seem like a trivial question but if we were to remove the testers from the equation (and replace them with developer-testers) how do we know that the testing performed by a developer is adequate and proves the service? Years of testing have given me trust issues and organisations generally don't have in place code-promotion paths that bypass the test area.

Have more questions but for the sake of brevity I'll save them for specific topics. Furthermore, I don't plan to answer these questions right away. I'm tackling them all at once at work and when I come up with, what I feel is the answer to a single question. I will put it up here to get some feedback and to share my decision.

Aside from all this I'll be writing and testing some services from a developer perspective. I'll trial the various SOA testing tools to gauge their effectiveness and I'll be having many discussions with my colleagues who are a mix of testers and developers.

Stay tuned