An evolutionary approach to implementing XP

I was rightly pulled up by David Anderson, author of the authoritative “KANBAN, Successful Evolutionary Change For Your Technology Business” for the following tweet

#Scrum and #Kanban are just 2 of many seeds for self organisation and continuous improvement. If you feed them with #XP they'll flourish

— Tom Howlett (@diaryofscrum) November 14, 2011

His point was that a wholesale methodology such as XP is at odds with Kanban’s evolutionary approach. Kanban gives us the tools to discover what’s impeding our process, where as XP prescribes an alternative. I replied that for us XP had been an evolutionary change so he suggested I write about it.

Engineers are creative. Engineers naturally want to discover new ideas themselves. When we tried to implement Extreme Programming (XP) back in 2003 it didn’t really happen, but we did at least become aware of the principles and practices involved. We took the first steps towards Continuous Integration, there was some rather half hearted Unit Testing going on and not much in the way of Pair Programming and that’s how it stayed for the next few years. All this time we were accruing more technical debt and the frustration from our managers that we couldn’t go faster was growing. This lead to a failed attempt to offshore some of our development. Things were looking pretty bad.

Some background

At Biomni we provide Service Catalogue and Request Management products to the Enterprise. Our main product has been continuously developed for over 10 years. In order to keep that product maintainable we have had to make a serious commitment to good engineering practice. We currently have a team of 7 developers 2 QA as well as a Development Manager and CTO who act as product owners.

The crucial change was autonomy

In 2007 I did my Certfied Scrum Master course with Jeff Sutherland and tried to introduce the principles and practices of Scrum to our team. We had a very successful pilot sprint and delivered much more than we had committed to. This led to the most important change of all. We were given autonomy. No longer were plans created behind closed doors but were created by us, collaboratively. We all started thinking more about the hows and whys of what we were doing. Rather than just letting grievances fester  or venting them to our development manager, we now had the power to do something about it. Thankfully the culture at Biomni had always been fairly honest and open so it didn’t take long for us to abopt our new roles as self organisers. Scrum provided us with a framework that facilitated collaboration. Sprint Plannings, Daily Scrums and Retrospectives were all full of ideas for change. Scrum’s fixed length sprints and the move to working with stories that were tested as soon as they were completed gave us the shorter feedback loops that accelerated change.

Through discovering ourselves what was generating waste we have steadily improved to the point that our practices now resemble the XP that we tried to adopt years ago. Here is a list of the main XP practices and how we have ended up adopting them:

Pair Programming

We have never enforced pair programming and sometimes don’t do it. But developing software well, not just making it work, is hard and pairing makes it much easier to get right. Over the years more of our team have become convinced by the benefit of pairing. Today its rare to work alone.

Planning Game

Our CTO liked to deliver a roadmap at the start of the release from this we created a product backlog of sorts. After a few releases he realised that this was prone too change and appreciated the 3 weekly sprint planning session when he could reprioritise and adjust the backlog based on what had changed. With our recent move to Kanban we may make these sessions even more frequent.

Where possible we include relevant business people in the team who have a particular interest in the feature or a close relationship with a customer who does. This has lead to getting the stories right and wasting less time producing something that isn’t wanted

Test Driven Development

It has become obvious to us all that keeping a large codebase maintainable and having the courage to change it requires a good suite of unit tests. TDD (Test Driven Development) is hard. It’s even harder to add tests to an existing codebase whilst maintaining a reasonable level of productivity and this what we had to do. Over the years we’ve learn’t more and more about making code testable and have finally reached a reasonable level of TDD practice. It’s still not perfect but the level of defects has dropped, we can change code without having to wonder what we’ve broken and our design is more decoupled.

Whole Team

Implementing stories you don’t believe in or don’t really understand in is at best soul destroying and generally a complete waste of time. We’ve found that having a business owner on the team makes a real difference to our understanding as well as enabling a truly collaborative and agile process . This has been a challenge because although we have autonomy within team we have no power to change what is outside it. We have always tried to bring relevant business people into the team but have struggled to get commitment from their bosses to give us their time. Still all is not lost, we do what we can and have seen some real benefit from it.

In many cases stories are our CTO’s brain child. These are the stories that move our product forward strategically and are no less important than those that come direct from users. Our CTO has learn’t in this situation to be part of our team and is never dicatatorial.

We have also bought our testers (QA) into the team although they prefer to keep some distance from being completely integrated we are able to help each other out and reduce a lot of waste. They now test a story as soon as possible after completion reducing feedback time and story lead time. This is one of the area I think were we can still make large improvements.

Continuous Process

In the dark years we would spend up to 30% of the total development time for a release on preparing it for delivery. This time was spent tackling a long line of defects in our code that were being found in the test phase and the building and testing of install packs. Over the next few years we completely eliminated both of these activities enabling us to reduce waste and have a more accurately predict our time to release (Which currently stands at about 2 weeks from the end of feature development).

Continuous Integration

All those random  tasks that needed doing before a release slowed our time to delivery and introduced defects right at the end of the cycle. First we documented the tasks then we automated them. Over the years our build tasks have grown in sophistication as we respond to new issues that arise. For example we know upgrade copies of customer database every night to get the earliest possible warning that something is awry with our scripts. Install packs have gone from being built once a release to being built on every check in. We fail early and fix it fast. This has given us real efficiency of delivery, to use the Kanban term. This automation has taken a considerable amount of time and one of our developers has made it his specialisation. Most importantly it leaves our developers to spend more time doing high value work.

Design Improvement

When working together poor design sticks out like a sore thumb. In pairs we have the courage to tackle fairly large rewrites whilst working on stories and it always seems to be the right decision to go ahead with a refactor. Last year we had a large area that used some legacy code. It complicated installations and being difficult to change stopped us improving a whole chunk of the product so the team rewrote the lot. It took about 6 weeks but the thought of it gone motivated us all. I had real doubts whether this was the right thing to do but the team was courageous and it paid off. We are lucky enough that between releasing a version and starting the next we are given a few weeks to work on whatever we want. This has proved to be a very productive time.

Small Releases

As David Anderson explains in the Kanban book releases have a cost not just to developers but to our customers too and our original plan of regular releases turned out to be too costly. Since our software is installed on customers servers they are reluctant to upgrade regularly and this means that if we release more often than customers are willing to upgrade we quickly accumulate a large number versions that need supporting concurrently. For this reason we have limited our releases to 3 or 4 a year.

Shared Understanding

Back in the dark days we attempted to come up with some coding standards but it seemed a bit BDUF (Big Design Up Front) and so we just left them to evolve. We try to keep ownership of our code as collective as possible to ease bottlenecks caused by only one developer being able to work in a specific area. The best way to achieve this seems to be for developers to swap pairs regularly, something that doesn’t seem to come naturally as developers naturally want to finish what they have begun.

Simple design means different things to different people depending on your level of expertise, through experience and sensible OO design we seem to have achieved a balance of code thats easily extensible yet simple enough to understand.

Programmer Welfare

We’ve been lucky to have a development manager who understands that pressure and overwork is counter productive. We get to work from home when we want to and all work at a sustainable pace.

Kanban

In recent months we have started cutting some of Scrums shackles such as fixed length sprints and using a Kanban board to better visualise how our process is working.

The biggest challenge this has revealed so far is that QA often become a bottleneck yet there are times when they have nothing to test. The obvious solution, apart from introducing a buffer, seems to be to even out story sizes and stop working in sprints where we try to have everything tested by the end of the sprint (giving them nothing to test at the beginning of the next one).

It’s early days I hope that by using the Kanban method we will be able to identify more ways to optimise our process and continue Kaizen.

Conclusion

So why did it have to be evolutionary? Why not subject the team to some sheep dip training and emerge the other side where we are now? Well I admit with some help it could have happened faster but would it have been accepted the same way? Would the culture of quality and continous improvement be as ingrained? Would we have really understood what we are doing? I honestly don’t know.

Tags: Agile, kanban, Scrum, xp