Since the late 19th century when Marconi started patenting other people’s ideas innovation stopped being collaborative and become competitive. The competitive culture spread top down, introducing formality and controlling methods to keep the top at the top and those at the bottom doing what they were told. In the 21st century agile teams are challenging the status quo and changing the way we think about work. For anything to grow it needs a friendly environment giving it the strength to fight off predators. In this post I look at the various environments we work in both real and in my dreams and assess which provide the sustenance for Agile to grow and spread.

Agile Team (real)

In traditional corporate culture when software projects fail Managers turn to the team in an attempt to find something or somebody to blame. When the smart team, who spend their time learning about better ways of doing their job rather than looking for people to blame, tell their manager they have a better way they are often given the opportunity to try. Motivated by this freedom it’s not hard to make an improvement. A smart organisation will take note of this and begin to learn from the underlings. A dumb one will protect their positions with lame excuses and continue as before until their business goes extinct and their smart developers move on. #limitedwin

Agile Collective/Cooperative (imaginary?)

A group of like-minded agile developers, read @ericries,  start a collective, build something great, disrupt a market, except no outside money, sell out, do it all over again. It’s the rock’n'roll of software development . Agile spreads by natural selection as they wipe out their competitors one by one #yesplease  (until the bigcorp realises whats happening and uses their huge muscle to crush them #horriblereality) 

Not very Agile Startups (very real)

Started by some opportunist entrepreneur and funded by short-term VC’s the pressure is huge. Agile dies a swift death. The startup will probably get sold to Tesco or meet some equally horrible fate #fail

Agile Corporation (maybe one day)

Big corp have the money to bring in smart people to help them beat the competition. They struggle to change but may just have the vision and determination to see it through. Unfortunately the divine right of the shareholder forces them to focus on short-term cost cutting and mass redundancies destroy the good work. It takes time to develop teams that collaborate well and this requires some stability #likelyfail

So where can we find happiness at work?

Well it’s a hostile environment out there for a fun-loving, segment disrupting agilista, but we’ve been fighting the status quo too long to give up. What options do we have to change our environment?

Revolution

Ok so lets build an army and take up arms against the evil fat cats? No? OK then #fail

Education

Lets replace Religious Education (R.E.) with the missing link in human evolution “Collaboration and Facilitation Education (C.a.F.E)”. Apart from improving the workplace, C.a.F.E could help in other important areas such as World Peace and Family Breakdowns #terawin. All the money saved could go toward building an enormous icon of Jeff Sutherland (see pic) #fail

Enough (Conclusion)

This post may be tongue-in-cheek but the problem is real. We need to keep things simple, the problem may be complex but the solution doesn’t have to be. Agile makes work satisfying because we become effective and brings happiness because we are fulfilling a human desire to collaborate. We only ask for a simple change that: Teams have a fundamental right to determine their own way of working and we can take it from there.

Who should plan?

People are motivated when they have self-direction. So by giving people self direction we see huge gains in effectiveness and creativity. Since working to other people’s plans kills our motivation we tend to avoid any project manager roles. In fact any role with the word “manager” in has bad connotations in the Agile world. Unfortunately its easier to change yourself than change your organisation so command and control management remains prevalent in most organisations despite killing effectiveness and leaving employees to soldier on waiting for a pay check.

Agile planning is done by the team who carry out the work because they are the only ones who understand what really needs to happen and have the best idea of the impediments that stand in their way. The team can collaborate to work out the best ways to work together and not tread on each others toes. A single project manager with a Gantt chart might not get such cooperation from his team.

Traditionally planning is left as the responsibility of a project manager. Project managers are measured on their ability to deliver on time and on budget after which they will move on to another project. This might be good for the project but delivering projects shouldn’t be the purpose of an organisation, it should be to create value. Agile teams shun individual targets in favour of looking at their effect on the wider system. We understand, with more than a passing interest in Systems Thinking, that individual targets increase competitiveness and reduce collaboration with the overall effect of killing effectiveness (yes we don’t think much of bi-annual appraisals either).

In Scrum we used to use the term “commit” when agreeing to what we would take on at the end of a sprint planning session. But even the work commit can be damaging. By committing to anything you close your mind to alternatives that will appear on the way. No we just say this is a plan and if it changes all the better or just regularly top up a backlog of things to do.

When should we plan?

Planning the future of an unpredictable project is pure waste, at best it will get thrown away at worse it may stop you delivering anything meaningful. As long as everyone understands the purpose of what you are trying to achieve it’s better to plan little and often (daily is a good starting point). I was going to say there are times when the simplicity and predictability of the thing you are doing mean you can plan at the beginning but I suspect you are just shooting yourself in the foot. Does anybody actually do it? Does it serve any purpose?

In Agile we may have some idea what we will do for the next sprint or iteration but the most important thing is the goal not any predictions of how we might achieve it. A daily standup allows us to at least give an intention of what we might do for the day and provides an opportunity to account for what we did yesterday. When the whole team  can see what the others are doing there is no need for someone to coordinate; we self organise. What if the people working on the project aren’t part of the same team? Simply put, you need to restructure your organisation into cross-functional self organising autonomous teams. I know this might not be achievable over night but the price you are paying for single function departments is too great to ignore.

How do you plan?

The greatest gains in our effectiveness can be seen from doing the right things, and our job is to discover what these things are. But is that planning or is that just part of the whole process?

So presuming we (the team) know roughly what we want to do, we give that thing a name (perhaps in the form of a user story), stick it up on a board where the whole team can see,  and as we become free we pick the tasks that are most in need of being “done” now and do them. Note that we “pull” the tasks we feel best positioned to do rather than have a project manager tell us. If you do this in a collaborative environment with all the people involved that’s all you need to do. While you are all together collaborating why not spend some of the time you are saving not being ineffectively productive by being a bit more creative? Have some fun, throw ideas around, you’ll be amazed at what you can do. All you need is a purpose.

I imagine this is going to  appear as bit of an anarchic rant to the average corporate project manager but perhaps there are some who can see the light. If it makes sense then take small steps. It will take generations to undo the damage that has been done by the management techniques of the industrial revolution. But if you just create one team who work like this you will have played a huge part.

Controlling

No you don’t control your process it is controlling you. Devised by a power-hungry manager, designed to ensure you conform and waste your time with endless documentation, your process defines your work. The only thinking required is when you try to fit a square peg into your round holed process. You care little for anyone else including your customer. Your greatest fear is that your process will change and those insecurities about the value of your work will be outed. Your fears are well founded, you’ll be easy fodder in the next round of redundancies.

Sycophantic

You go gooey eyed at the mention of it and can find no fault in your beloved process. You preach with religious fervor to family, friends although probably not to your colleagues who may be a little cynical and prefer to just get on with delivering stuff. It’s unlikely that you are really doing what you believe in although you may be suffering from denial and be conveniently ignoring the bits of your process that required real change. Your devotion may have been triggered by an early experiment the results of which you still cling to. You believe that if you follow the process good things will happen but your missing any real purpose.

Promiscuous

You say “Process X was so last decade! Y is where it’s at, all the Thought Leaders are saying it so it must be true”. Once you realise Y isn’t really working you can jump on the Z bandwagon. You’ve never stuck with anything long enough to get a real understanding of it and like the Sycophant care more about the process than its purpose.

Healthy

Although your process may have been used by others before you’ve made it your own. You share the same principles and have a deep understanding of them. You give each other the freedom to make new discoveries. You’re both idealistic and pragmatic. You have a little poster of the agile manifesto by your desk to remind yourself what its all really about: “Individuals and interactions over processes and tools”.

How do we approach what lies beyond? Our ideology doesn’t cut it in this world but stories just might. When opportunities for small practical steps arise we need courage to step up and take them. Who knows where it might lead?

We decided to have a retrospective focused exclusively on the collaboration (or lack of) between developers and testers in the hope that we could identify areas where we could make both of our lives simpler, reduce waste and lead time.

First question: Do we only consider our own function rather than the best way to get the story to DONE?

I asked the team to write down a score ranging from 0 if you only consider your own function to 10 if you always consider what is best for the whole process. Answers ranged from 3-8 and we each gave a brief reason for why we scored the way we did.

Next question:  What is the best way for QA to discover what needs testing?

We all agreed that getting the developers who developed the story and the tester that was going to QA it together before testing begins, to show what has been done and discuss the risks would be best. This will also provide an opportunity to discuss the unit tests we have done. We’ll also try to involve QA more in the UX design process.

Next question: How can we shorten the time from end of functional development to release?

I should have worded this one differently, suggesting shortening the time put everyone on the defensive. “Make best use of” would have been much better. Currently we have a period of about 2 weeks from the end of feature development to release. But with the progress we have made automating installs the things we need to test during these periods have completely changed. We’ve agreed to get together before the next release to plan this further.

Last Question: What should happen if a change requires a large regression test?

If developers make a sweeping change at a level that effects the whole product should we ask QA to do a complete regression? We would have a set of automated regression tests but its not something we’ve managed yet. When this happened recently very close to release all the developers spent the day doing exploratory testing and we all agreed this was really useful and a good learning experience for all.

So how did it go? Well we identified lots of areas where we can improve our process by collaborating more. Only time will tell how well we adopt the new ideas. The preprepared questions to discuss in the retrospective seemed to work well and kept the dialogue focused. We usually do retrospectives in the office but this one was remote with the questions on shared slides, I don’t think too much was lost because of this. What do you think?

Requests for help outside our day-to-day development usually come via email. Now I’m not a big fan of emails. I sit and stare at them wondering what useful information I can glean. To make matters worse the problems have often been around the houses by the time it gets to development and there is a long chain of discussion.

This has happened because it’s often not easy to work out who really knows how to fix a problem. I find the best strategy in this situation is to jump on Skype and chat to the person who originally reported the issue and start all over again pulling on other people as required, it may be a bit disruptive but problems can be solved in minutes rather than days reducing work in progress and allowing everyone to get on with more productive work.

I guess this is a form of Swarming. Swarming is a term used in Kanban where the whole team attempts to solve a blocking issue quickly to keep the tasks flowing. Although it may appear disruptive in my experience it is often worth it. If you haven’t got anything to offer drop out of the swarm. The difference here is the people in the swarm aren’t on the same team or even the same department and don’t usually have the chance to collaborate.

The earlier we deal with these blockers the better. Perhaps if we had regular cross departmental collaboration we could eliminate many of these tricky issues altogether. A standup where a representative from each department could quickly describe what they are working on and any impediments they face could trigger collaboration that could save many painful hours of work.

And what about the side effects of all this collaboration? Shared learning and understanding? New ideas and creativity? Less disparity between what the customer wants and the developers build? An end to departmental silos? What do you think?

#Scrum and #Kanban are just 2 of many seeds for self organisation and continuous improvement. If you feed them with #XP they'll flourish—
Tom Howlett (@diaryofscrum) November 14, 2011

His point was that a wholesale methodology such as XP is at odds with Kanban’s evolutionary approach. Kanban gives us the tools to discover what’s impeding our process, where as XP prescribes an alternative. I replied that for us XP had been an evolutionary change so he suggested I write about it.

Engineers are creative. Engineers naturally want to discover new ideas themselves. When we tried to implement Extreme Programming (XP) back in 2003 it didn’t really happen, but we did at least become aware of the principles and practices involved. We took the first steps towards Continuous Integration, there was some rather half hearted Unit Testing going on and not much in the way of Pair Programming and that’s how it stayed for the next few years. All this time we were accruing more technical debt and the frustration from our managers that we couldn’t go faster was growing. This lead to a failed attempt to offshore some of our development. Things were looking pretty bad.

Some background

At Biomni we provide Service Catalogue and Request Management products to the Enterprise. Our main product has been continuously developed for over 10 years. In order to keep that product maintainable we have had to make a serious commitment to good engineering practice. We currently have a team of 7 developers 2 QA as well as a Development Manager and CTO who act as product owners.

The crucial change was autonomy

In 2007 I did my Certfied Scrum Master course with Jeff Sutherland and tried to introduce the principles and practices of Scrum to our team. We had a very successful pilot sprint and delivered much more than we had committed to. This led to the most important change of all. We were given autonomy. No longer were plans created behind closed doors but were created by us, collaboratively. We all started thinking more about the hows and whys of what we were doing. Rather than just letting grievances fester  or venting them to our development manager, we now had the power to do something about it. Thankfully the culture at Biomni had always been fairly honest and open so it didn’t take long for us to abopt our new roles as self organisers. Scrum provided us with a framework that facilitated collaboration. Sprint Plannings, Daily Scrums and Retrospectives were all full of ideas for change. Scrum’s fixed length sprints and the move to working with stories that were tested as soon as they were completed gave us the shorter feedback loops that accelerated change.

Through discovering ourselves what was generating waste we have steadily improved to the point that our practices now resemble the XP that we tried to adopt years ago. Here is a list of the main XP practices and how we have ended up adopting them:

Pair Programming

We have never enforced pair programming and sometimes don’t do it. But developing software well, not just making it work, is hard and pairing makes it much easier to get right. Over the years more of our team have become convinced by the benefit of pairing. Today its rare to work alone.

Planning Game

Our CTO liked to deliver a roadmap at the start of the release from this we created a product backlog of sorts. After a few releases he realised that this was prone too change and appreciated the 3 weekly sprint planning session when he could reprioritise and adjust the backlog based on what had changed. With our recent move to Kanban we may make these sessions even more frequent.

Where possible we include relevant business people in the team who have a particular interest in the feature or a close relationship with a customer who does. This has lead to getting the stories right and wasting less time producing something that isn’t wanted

Test Driven Development

It has become obvious to us all that keeping a large codebase maintainable and having the courage to change it requires a good suite of unit tests. TDD (Test Driven Development) is hard. It’s even harder to add tests to an existing codebase whilst maintaining a reasonable level of productivity and this what we had to do. Over the years we’ve learn’t more and more about making code testable and have finally reached a reasonable level of TDD practice. It’s still not perfect but the level of defects has dropped, we can change code without having to wonder what we’ve broken and our design is more decoupled.

Whole Team

Implementing stories you don’t believe in or don’t really understand in is at best soul destroying and generally a complete waste of time. We’ve found that having a business owner on the team makes a real difference to our understanding as well as enabling a truly collaborative and agile process . This has been a challenge because although we have autonomy within team we have no power to change what is outside it. We have always tried to bring relevant business people into the team but have struggled to get commitment from their bosses to give us their time. Still all is not lost, we do what we can and have seen some real benefit from it.

In many cases stories are our CTO’s brain child. These are the stories that move our product forward strategically and are no less important than those that come direct from users. Our CTO has learn’t in this situation to be part of our team and is never dicatatorial.

We have also bought our testers (QA) into the team although they prefer to keep some distance from being completely integrated we are able to help each other out and reduce a lot of waste. They now test a story as soon as possible after completion reducing feedback time and story lead time. This is one of the area I think were we can still make large improvements.

Continuous Process

In the dark years we would spend up to 30% of the total development time for a release on preparing it for delivery. This time was spent tackling a long line of defects in our code that were being found in the test phase and the building and testing of install packs. Over the next few years we completely eliminated both of these activities enabling us to reduce waste and have a more accurately predict our time to release (Which currently stands at about 2 weeks from the end of feature development).

Continuous Integration

All those random  tasks that needed doing before a release slowed our time to delivery and introduced defects right at the end of the cycle. First we documented the tasks then we automated them. Over the years our build tasks have grown in sophistication as we respond to new issues that arise. For example we know upgrade copies of customer database every night to get the earliest possible warning that something is awry with our scripts. Install packs have gone from being built once a release to being built on every check in. We fail early and fix it fast. This has given us real efficiency of delivery, to use the Kanban term. This automation has taken a considerable amount of time and one of our developers has made it his specialisation. Most importantly it leaves our developers to spend more time doing high value work.

Design Improvement

When working together poor design sticks out like a sore thumb. In pairs we have the courage to tackle fairly large rewrites whilst working on stories and it always seems to be the right decision to go ahead with a refactor. Last year we had a large area that used some legacy code. It complicated installations and being difficult to change stopped us improving a whole chunk of the product so the team rewrote the lot. It took about 6 weeks but the thought of it gone motivated us all. I had real doubts whether this was the right thing to do but the team was courageous and it paid off. We are lucky enough that between releasing a version and starting the next we are given a few weeks to work on whatever we want. This has proved to be a very productive time.

Small Releases

As David Anderson explains in the Kanban book releases have a cost not just to developers but to our customers too and our original plan of regular releases turned out to be too costly. Since our software is installed on customers servers they are reluctant to upgrade regularly and this means that if we release more often than customers are willing to upgrade we quickly accumulate a large number versions that need supporting concurrently. For this reason we have limited our releases to 3 or 4 a year.

Shared Understanding

Back in the dark days we attempted to come up with some coding standards but it seemed a bit BDUF (Big Design Up Front) and so we just left them to evolve. We try to keep ownership of our code as collective as possible to ease bottlenecks caused by only one developer being able to work in a specific area. The best way to achieve this seems to be for developers to swap pairs regularly, something that doesn’t seem to come naturally as developers naturally want to finish what they have begun.

Simple design means different things to different people depending on your level of expertise, through experience and sensible OO design we seem to have achieved a balance of code thats easily extensible yet simple enough to understand.

Programmer Welfare

We’ve been lucky to have a development manager who understands that pressure and overwork is counter productive. We get to work from home when we want to and all work at a sustainable pace.

Kanban

In recent months we have started cutting some of Scrums shackles such as fixed length sprints and using a Kanban board to better visualise how our process is working.

The biggest challenge this has revealed so far is that QA often become a bottleneck yet there are times when they have nothing to test. The obvious solution, apart from introducing a buffer, seems to be to even out story sizes and stop working in sprints where we try to have everything tested by the end of the sprint (giving them nothing to test at the beginning of the next one).

It’s early days I hope that by using the Kanban method we will be able to identify more ways to optimise our process and continue Kaizen.

Conclusion

So why did it have to be evolutionary? Why not subject the team to some sheep dip training and emerge the other side where we are now? Well I admit with some help it could have happened faster but would it have been accepted the same way? Would the culture of quality and continous improvement be as ingrained? Would we have really understood what we are doing? I honestly don’t know.

From my point of view as Scrummaster the most important thing was the rythm of regular delivery and maintaining the trust and autonomy we had established. Mindful of the good engineering practices encouraged in scrum we attempted to at least never increase technical debt however the continual pressure to deliver what we committed to discouraged any major refactorings. These were saved for interim periods that we tended to have every few months whilst the product was released, by which time the list of things to do was big.

Over the years we have managed to reduce our technical debt, engineering standards are high and defects low. Reduced technical debt makes it easier to focus on adding great functionality and we can really see the benefits it has bought. Constantly reducing technical debt means as the software grows more complex our velocity doesn’t reduce, it may even increase, although we shouldn’t rely on velocity as a measure of productivity.

The more we collaborate the more we care about the code the rest of the team writes. Pair programming is the practice that has helped us reduce technical debt the most, so many times I’ve been stopped from doing something quick and dirty by a team mate. It’s like having an angel looking over your shoulder!

The question is if we hadn’t introduced Scrum would we have even less technical debt and the greater productivity that brings? Using Kanban to improve our process has proved an interesting alternative and has freed us up to spend more time reducing technical debt as needed. However it’s the XP practices such as pairing and the feeling of common code ownership that set us on the path of reducing technical debt and it’s Scrum that gave us the autonomy and collaboration that allowed us to commit to these practices so I can safely say that despite the issues for us Scum helped us reduce technical debt.

With Scrum came a self organising team. Our managers quickly realised that we were capable of making development related decisions ourselves. I think it was one of the big boosts to our productivity as we continually honed our practices and gained satisfaction from continual improvement. My only frustration is that we haven’t been able to spread this outside the development team.

When making decisions within a team its best to use the highest bandwidth possible, face to face or at least a conference call are necessary. Email is too painful. Where it’s just not possible to get everyone together or you really don’t know who it’s best to consult we can use an online tool. I’m interested to hear how the new private social networking tools can help with this?

Managers resist having too many people involved in a decision in the fear that it will complicate the process. But the more relevant committed people involved the more likely they are to come up with the best solution quickly. It’s people who aren’t directly  affected by the decision but with something to prove that complicate it. These are the people who gave meetings such a bad name.

At a company level it’s not  practical to make decisions in collaboration with all interested parties (although perhaps it is using online tools) so it’s important to communicate the decision and the reasons for the decision well.  Taking care when making decisions helps everyone feel committed to a shared cause, motivated and with a good understanding of goals.