Questions from my Continuous Delivery Talk

My short talk on how we do Continuous Delivery at 7digital generated many questions from both the audiences at Devs in the ‘ditch and London ACCU.  Also, a couple more were asked on Twitter after the events.  Here are are the ones I can remember and my answers.  If anyone has any more questions please add them to the comments.

Can you choose which release version to deploy?

As we deliver web-based services, not products, we are always aiming to release the latest version which is almost always the of HEAD of the Master/main integration branch merged into a Release branch.

We rely heavily on TeamCity to trigger our deployments as well as our continuous integration.  It includes a feature called ‘pinning a build‘, which prevents it or it’s artifacts from being deleted in a clean-up policy.  It also allows us to reference these artifacts in another build, such as a deployment build.

Once the Release branch has been updated with the changes in the HEAD of the Master branch, and all of the tests have passed and we are happy, the build is ‘pinned’ in TeamCity and we kick off the Deploy to Live  build which picks up the pinned artifacts and deploys them to the production servers.

We can choose what build should be pinned and therefore what artifacts are released to Live.  We don’t necessarily version our releases because we never refer back to the versions and only a single version is ever live at one time.

How do you do a rollback?

We ‘un-pin’ the build and artifacts of the ‘bad’ release, ‘re-pin’ the build and artifacts of the previously known ‘good’ release and run the Deploy to Live step once again.  This effectively does a ‘roll forward’ with known good artifacts.

What role do QA have in the process and do they trust the automation?

QA are generally involved throughout the process.  Together with the Developers we will both fulfill the traditional role of a BA and analyse a piece of work, creating Acceptance Criteria which normally form the basis of the Automated Acceptance Tests.  Also, this means that QA are fully versed in the feature or change when it comes to UAT and explanatory testing and together we can make a judgement call as to whether a change actually needs QA manual testing or is sufficiently covered by the automated tests.  Being involved all the way through gives them confidence in the process.

A point to make is that we don’t have a QA Team as such, each development team includes a QA person and a Product Manager.  We all sit together and attend the daily stand-up so everyone is aware of what is taking place, the mood of a change and can jump in at any point.

How do you handle large features/pieces of work?

We hold an analysis session within the team, including the developers, QA and Product Manager to break down the work into as small a user story as possible, aiming for under a day.  Each story needs to be a single contained piece of the functionality which can be released on it’s own.  This is not always possible and in these times we employ Feature Toggles which will hide a feature until it is ready.

What we don’t do is have feature branches.  This is something that must be avoided to ensure that we are always integrating all changes and any problems are highlighted as early as possible in the development cycle.

What about database schema changes?

We use a tool we developed internally, but have since Open Sourced: DBMigraine.  There are a couple of blog posts on the 7digital Dev Blog here and here which explain it in more detail, but in essence it builds databases from a set of creation scripts applies migration scripts, and performs consistency checks between databases.

Using this tool we build a database from the scripts and migrations at the beginning of each Integration test suite and run the tests against the new schema.  This should hopefully flag up any major problems before these migrations are also applied to the Systest and UAT databases which are integration points for all of our apps sharing the same database.

It’s worth noting that we try to avoid destructive migrations, but this process has allowed us to gradually delete unused tables in a tested manner.

Edit – new Question from @AgileSteveSmith

What cycle time reductions have you seen?

In my reply, I linked Steve to the following two posts on the 7digital Developers’ Blog related to productivity at 7digital: “Productivity = Throughput and Cycle Time” & “Development Team Productivity at 7digital“.

The posts illustrate, with data tracked from our own work items, that there was an incredible reduction of Cycle Time in over the course of 2009 to 2011 – you can even see the massive spike at one point where things got worse before they got better, as I mentioned in my presentation!

A full report was put together, with even more charts and graphs, which can be downloaded from the second blog post.

Continuous Delivery at 7digital

It began with an off-hand comment on the ACCU General mailing list that at 7digital we release on average 50 times per week, across all of our products.  I thought nothing of it, virtually all of our products are web-based, which makes it relatively easy to deploy on a regular basis, but it seemed that others were interested in how we do it and so I was cajoled into giving my first presentation.

I began by explaining what we understand as Continuous Delivery – a set of repeatable, reliable steps which a change must go through before being released.  In our case most of these steps are automated.

I described where we came from and how we approached the change, in both a technical and business manner, and where we would like to see ourselves going.  I then included a flowchart of  A Day in the Life of a Change at 7digital, which Prezi allows me to ‘bounce’ around as we hit different stages in the pipeline.

I answered many questions clarifying how we handle rolling back a bad release (we actually ‘roll-forward’ with artifacts of the previous ‘good’ release), whether our QA team are comfortable with the process (yes, they help write the criteria),  and how large pieces of work are tackled (we try to break them down into deployable pieces no bigger than a day).

Here are the slides:

One Test or Two?

Paul Grenyer & Chris O’Dell

“I know what you’re thinking: “Is that two tests or only one.” Well, to tell the truth, in all this agility, I’m not sure myself. But this is Test Driven Development, the most power development technique in the world, and could give you clean code, you’ve got to ask yourself one question: Do I feel expressive? Well, do ya, punk?”

Paul: I have a Windows Presentation Foundation (WPF) application that loads a list of widget names from a database and uses them to populate a drop down box. There is no default widget, so the user must select one from the list and click the load button to load it. The behavior is such that before a widget name is selected the button is disabled and cannot be clicked.

One pattern often employed in WPF applications is Model-View-ViewModel. The view model can use variations of the Command Pattern for processing commands from and giving feedback, such as the enabled state of a button, to the View. MVVM is intended to give separation between a view and its logic. Therefore the view model is usually devoid of User Interface (UI) code and easy to instantiate and run unit tests against.

I have a unit test that instantiates a view model and checks that the load command cannot be fired until a widget name is selected. A simplified version is show below, without the view model instantiation:

public void testThatExecuteIsOnlyPossibleIfNameIsSet()
 model.Name = "Something";

Recently I have also been involved with the ACCU Mentored Developers project based around Growing Object Orientated Software Guided By Tests by Steve Freeman and Nat Pryce. In this book they write a lot about the naming and granularity of tests. So I posted the above code to the list and asked what the project members thought: “Should the above test be one test or two?”.

Chris: Without blinking I immediately replied that the above should definitely be split into two distinct tests so that a failure would be obvious from the test name. The above test has two asserts and as such either of these could fail.

Paul: Chris, and all the others who replied are of course right on the money (no one suggested it should only be one test), but something didn’t sit quite right with me. In this small example the extra code of two tests is not very much: the method definition, a pair of curly braces and some spacing. The problem for me is that test classes should not be treated that much differently to normal classes and any class with with a large number of methods becomes difficult to maintain and difficult to even to navigate. Although modern IDEs do make this easier with regions (C#) and code collapsing (Eclipse).

Chris: I firmly believe that it is worth the extra code – as many others also remarked, the five minutes now and extra curly braces could easily save twice that if you need to hunt down a related bug in future.

I went on to explain that in terms of application logic the two scenarios “with a name” and “without a name” are expected to be mutually exclusive and the two tests will ensure this by isolating each scenario and addressing them individually – with explicit test names.

In regards to the large classes this can be tackled by breaking your tests down into smaller classes, generally per scenario. For example, group all of the positive “happy path” tests together into one class and all the negative, error handling tests into another class and store both classes in a folder given the name of the class under test.

Paul: Again, Chris is of course right. Although the implication that a test method should only have a single assert is a whole other discussion.

Then Nat Pryce came along with another suggestion altogether:

“To really play Devil’s advocate, I think there should be THREE tests!

1. The model initially has no name

2. The model cannot execute when it has no name

3. The model can execute when the name has been set.

You could squeeze that into two:

1. The model initially cannot execute

2. The model can execute when the name has been set

But I think the state transitions and constraints are clearer with three tests.”

Even with there now being the extra code for three functions, rather than one, this is of course the answer. It even satisfied the idea of one assert per method. The most difficult idea for me was not testing preconditions specific to a test if they were already tested in a another test .I have consequently modified my way of thinking and a lot of my test code.

Originally published in CVu July 2011.

Experiences of Pair Programming

The following was printed in the January edition of the CVu journal from the ACCU, which is a great organisation of programmers that I would encourage you to look into.

In the last CVU Editorial Steve asked what pair programming is like as he had had limited experience, usually on one-off tasks lasting no more than an hour at a time and in all cases he’s found the required level of concentration to be extremely high – and it is. Three months ago I joined 7digital, a leading digital media delivery company, who’s development team use an agile approach including daily stand-ups, TDD and kanban boards, plus pair programming. In fact, the whole team pair programs as much of the time as is possible.
As Steve found out it can be quite exhausting, and I certainly discovered this after my first couple of weeks with this approach. We hold regular 1-2-1 meetings with the Development Lead and I mentioned this to him in my first meeting. He agreed, pair programming is indeed intensive, and it is counter-productive to be 100% focussed at all times, frequent breaks were recommended and the pair will generally break for 10 minutes returning to their respective machines to catch up on office email, read a little Twitter and some blogs before jumping straight back in ready to focus once again. I would say it is similar to the concentration techniques out there such as Pomodoro, whereby you focus solidly for a set amount of time, then break, which is an interesting technique that I have tried in the past and actually found myself to be achieving more.
One other thing I brought up in my 1-2-1 was that I tended to take a more ‘passive’ role in the pairing – watching and commenting rather than coding. As I was new to the code base this felt like the right approach to take, but as is always the case, the learning does not sink in until you actually type the code yourself. One method to increase my participation was a ‘ping-pong’ pair programming approach, whereby one developer writes a failing unit test and the other developer writes the code for it to pass, which fits in excellently with our TDD approach. But, all in all I needed to work on being more assertive. As my Lead had stated, pair programming is a skill just like any other and it must be learned and honed for the benefits to be reaped.
Also on the personal side, pair programming got me up to speed with the code base extremely quickly, and with pairs swapping once or twice a week, I got an overview of many parts of the system far quicker than if I’d been programming alone on a single feature. I also got to know the team which is a massive bonus as there is so much to be learned from your colleagues.
On the business side, we have reported a huge decrease in bugs reaching production and thereby a marked increase in customer satisfaction. The ‘second pair of eyes’ which your partner provides is hugely valuable – how many times have you found yourself going in circles trying to track a bug down only to find it was a silly typo or something just as obvious? Your pair is your saviour in these situations and they more often than not point out the mistake before it gets to compilation and nowhere near production.
The changing of pairs may sound chaotic, but what usually happens is is that one person ‘owns’ the story and partners switch around. The advantage of this is knowledge transfer. Of course, at first there is a small part of needing to bring the new partner up to speed, but then they can jump straight into assisting with development. If it takes too long to bring someone up to speed, then your task is too big or your approach is too wieldy, either way it needs to be reassessed, which you can do with your partner. This hands-on approach to knowledge transfer is far more effective than on the oft arcane “handover document” usually written just before you head on a 3 week holiday… As an example there was a piece of work to allow for gift card purchases, broken down into many stories but with one overall Minimum Marketable Feature (MMF) and a colleague ‘owned’ this piece of work with at least 3 other developers having paired with him on it and I was one of them. He took a weeks holiday during the development of the MMF and for that week I took over the ownership of the work pairing with one of the other guys and we continued it through to near completion without a hiccup nor a “handover document” in sight. From a business point of view the work did not need to stop or even slow down noticeably while he was away and the knowledge of the implementation is spread between at least 4 people. From the Dev’s point of view he got to take holiday when he wanted and without rushing together a handover on the afternoon of his last working day.
As with all methodologies it is not a silver bullet and relies heavily on the attitudes of the developers involved. If anyone prefers to work alone – building empires of arcane code, then they will find ways to continue to do so and make their pair miserable in the process. In fact, there’s an excellent list of 10 ways to Kill Pair Programming, which I suggest you read – and do the complete opposite of everything stated.
Your pair is your reviewer, your counsel, your sounding board, your mentor, your student and the person keeping you focussed. There is one thing though that I do miss from coding alone, and that is listening to music with my headphones on, but I can happily trade this for the benefits listed above.

Published in the CVu

I have been a member of the ACCU (Wikipedia article) for a couple of months now and it’s an excellent group of helpful and knowledgeable people with two fantastic journals – CVu which is members only and Overload –  and I managed to get an article printed in the January edition of CVu.  It was my first printed article and I’m both extremely happy and shy about it.

The article is a direct response from the Editor Steve Love’s desire to know more about Pair Programming and so, I sent in a personal account of my experiences.  If you are a member and receive CVu, please could you read it and post your comments back here for me as I would appreciate feedback.

The group mostly focus on C and C++, and although I’m a C# .Net developer by day, I felt that exposure to people in other realms is essential to the growth and rounding of a developer’s knowledge. So, despite the fact that the majority of the code related discussions go well past me, the discussions on professionalism, self improvement, team work and so on, are language agnostic and extremely enlightening.

Please, go and take a look,  join their general mailing list (which is open to non-members as well as members)  and consider joining.