Agile Development, The Beach Boys and Murry Wilson’s Mixing Desk

In the early 2010s, I was hired for the perfect one-year freelance gig. It was a well-conceived and innovative idea for a product, with real social value. They wanted a CTO who could make technological decisions, represent the team’s technological decisions, and to provide an interface with the parent organisation, whose own technology team had a very different methodology. The project started with three months of weekly workshops, then during the six months of product development, I worked half time. Just after the launch, we appointed my successor, and I went back to three months of weekly meetings while everything stabilised.

“CTO” can mean different things. At one end of the scale it can refer to a start-up cofounder who writes code, and at the other, an executive, who oversees the procurement and delivery of technology products and services. I don’t routinely fill either of those roles, but I do most of the bits in between, and usually enjoy the work. My technology input is more whiteboard than code, and my executive decisions are usually verbal, and light on bureaucracy.

I don’t like jobs where there is no new technology being developed, but it doesn’t have to be me leading the innovation. Unlike many ex-coders, I also get satisfaction from representing the technology function within an organisation to all kinds of stakeholders, including customers, and working out what is needed from technology, and what might be involved in getting it in place.

So, this dream contract involved quite a bit of whiteboard stuff, and a lot of acting as an enabler for a really great team. In retrospect, the year’s work was a success: a world-class product was launched; a sustainable team was in place (expensive contract staff having been replaced by permanent staff), and our lead developer was promoted to CTO, and doing a great job. However, it was the first time I’d had a job, which was perfect on paper, but at which I didn’t really enjoy either the whiteboarding or the enabling.

The whiteboard part would have been fine, if the team had needed me, and it would be unfair on them to say that I didn’t enjoy any of the time spent with them. The only problem for me is that the product could have been built perfectly successfully without my creative contributions to its designs. However, if I hadn’t applied myself wholeheartedly to the other role, I can safely say that it wouldn’t have been built at all.

I have seen a lot of systemic problems and a lot of office politics, with meddlers who deliberately get in the way, without contributing anything. This however was such an extreme, that it required an exhausting and expensive effort of the most subtle political manoeuvring imaginable. If I’d wanted to do that for a living, I would be working in Whitehall, or at least, would never have left the public sector.

This is the situation I found:

• The product team were funded, recruited (though expensively) and ready to deliver
• The parent organisation’s world-class subject-matter experts were on-hand to advise
• The team negtiotiating partnerships were creating consensus despite every partner originally wanting special treatment

However

• A team had been put in place to oversee the project, who took it upon themselves to object to, disrupt and in some cases, completely obstruct, every aspect of what the team succeeded in doing anyway
• Despite having a team whose members had won dozens of national and international awards, creating products with global reach, the board put in place a team of parochial jobsworths, to oversee work they barely understood, to the extent of our requiring their approval for every decision

I am proud that we delivered a great product, and I am even proud of the deviousness with which I protected the project. However, I am ashamed of the personal limitations, which prevented me (and other executives) from making those in ultimate control see sense, and am still bewildered by the fact that success meant spending time, effort and hard cash on completely pointless activities.

Naming names

I held this essay back because it implicates some people, my experience of whom was extremely negative, but who nonetheless do not deserve a grenade being thrown at their careers.

I am hoping that there will be enough fun and games in the anecdotes, to keep the audience amused without the names, and I just can’t face a flame war.

PHASE ONE: First signs of obstinate box-ticking

I was told before signing the contract, that the product would be created by a new company, wholly owned by a larger organisation, whose technology team would not be directly involved, but who would have some kind of role, providing quality assurance. Given the awe-inspiring track-record of the new team, I wondered why this was necessary.

The only reason I could find was a historical accident. There had been a period in the development of the idea, during which it had been assumed that the incumbant team would develop the product. Having later been denied this privilege, they were tossed a bone, and given some kind of role. Whatever the truth of the matter, and amateur mind-reading aside, they certainly acted as though they had an interest in this decision being discredited, so — perhaps indirectly and only in part — they wanted the project to fail.

The incoming team had already decided that the product development would follow orthodox Scrum methods, and that if a web application was developed, it would be done using Ruby-on-Rails. Any thoughts of app development, would be left until the product was market-tested in its launch format, a responsive, mobile-friendly website. One tiny contribution I did make, was to discourage the team from going with mobile-first design, because all the early evidence from user research, pointed to a strong preference for laptop-based sessions.

These early decisions seemed entirely reasonable, for the kind of thing we were building — a social application with a lot of content delivery and user-submissions. While I had some reservations about Ruby-on-Rails’ one-size-fits-architecture, I was quickly impressed by the speed of iteration, and the team convinced me that iterating the user experience faster, was ultimately more valuable than — for instance — spending time solving scalability problems which might not occur.

Given that my job was to manage the design and build of a completely new product, the first sign of a problem, was that I was required to provide a system diagram before we started work. I reduced it to a bunch of necessary functions, and lists of candidate technologies which could perform them. It was eventually accepted, but with a lot of reservations.

PHASE TWO: “Bureaucracy is the death of all sound work”, Albert Einstein

As we started work, I was called into more and more meetings about how the governance of the development would be performed. When ITIL was suggested, I provided an analysis of how the methodology did not map onto our actual processes, so the documents created and tests performed, would not represent our plans or assess any risks accurately. This was before version 4, which to some extent, accepts the advantages of Agile practices. Decisions in Agile working, are taken only when enough evidence has been gathered, which is rarely right at the start of a new development.

There are millions of web-pages describing Agile, but suffice to say, in traditional software engineering, before a line of production code is written, highly granular details of precisely how the objectives will be achieved would already have been documented. These days it is normal to be clear on what the objectives are, but to work out precisely how to achieve them, by starting with the simplest possible working system, and proceeding using the sound assumption that actual user behaviour is a better guide than what can be achieved by requirements analysis alone.

However, despite being (though I say so myself) quite articulate on the point, it turned out that this was not a negotiation, but a stipulation. So, despite not being fit for purpose, we were forced to apply a methodology, which required us to deliver 157 documents, which served no purpose, other than determining whether we had filled them out correctly. This was not easy, and many needed multiple attempts to pass muster.

I decided that the best thing I could do to help the project succeed, was to avoid letting ITIL document authoring distract the team. Anyone who knows me, will also know that I was no more suited to this task than anyone else, so what to do?

Around this time, I had got back in touch with an old school friend and bandmate, who was looking for a career change. He was just as dissolute as I was as a school student, but somehow we had both become quite hardworking in our corporate lives, and he had spent quarter of a century working in regulated industries, where hyper-documentation is the order of the day. From a graduate job in nuclear engineering to a career in public telephony infrastructure, he has written documentation to specs, which are much more complex than systems to which they refer.

I explained the situation to our CEO, and got the budget in place, and he set to work.

I put in place some strict rules:

• He would fill in the documents, interrupting the team as little as possible
• The team would help him whenever asked, knowing that he would only ask them when absolutely necessary. I explained the alternative to them, which made this completely unproblematic. While mystified at what they were being asked for, they were polite and helpful, even when breaking the flow of heads-down coding.
• To speed things up, I would countersign my approval of all documents without reading them.
• When in meetings arising from the inevitable claims that there were irregularities, I would defend the documents to the Nth degree, but if necessary, authorise re-authoring.

There was not a single document which was accepted without issues. Once corrected, they were filed away, never to be read again. It is unclear how this makes the system more effective, less unreliable, more secure etc. It explains a lot about public sector IT systems and the many problems of exceeding budgets, deployment failures from the NHS to the BBC.

The bureaucrat-in-chief had a particular style. He could not possibly have read the documents in time for some of the meetings, but he always ensured that he had been briefed about a problem in a subclause of a subclause, in order to have something to throw back at us. However, we politely re-authored, and everything passed. Sometimes we had to give them a couple of weeks to read a document, which we had been given only a day or two to produce. It was quite quite insane.

CODA: What does this have to do with the Beach Boys?

When the Beach Boys started out, Murry Wilson, father of Brian, Denis and Carl, and uncle of Mike Love, was their producer. He had been responsible for teaching them the discipline of singing in harmony. And, discipline was the word, as he had a bad temper, to the point of sometimes being violent. Brian soon emerged as one of the most talented record producers of the era, and took over. However, they were so afraid of Murry, that they didn’t know how to tell him.

When Brian had things the way he wanted, Murry would just intervene, tinkering with the controls. So, what to do?

The answer came from their engineer, Chuck Britz, who built a mixing console which Murry could play with, without having any effect on the final output, and allowing the band to get on with their work. My document-savvy friend was our Chuck Britz, and the approved ITIL documents were like Murry Wilson’s mixing desk.

EPILOGUE: Little Green Men

After all the documents had been accepted, there were still a couple of items left in the ITIL process. One of these was a disaster recover exercise. I asked the bureaucrat-in-chief what this involved, to be told that we needed to kill the main server while running, and go through the process of getting the system up and running again.

At our next meeting, we explained that in our cloud-based model, everything was designed to be fault-tolerant, and a new server would spring into life. I asked whether that would do. They said that they had to witness this experiment, so I said that we could do it there and then, and that it would only take a minute or two.

In front of him, and two of his staff, we killed a server, and tested the site, to ensure that it was down. We watched the terminal output, as a new server booted up, and then tested the site again. So, I asked whether that section was now complete. He said that one of his staff had to write it up, and send it to him. I asked whether this could be done from his phone there and then, but apparently, the appropriate MS Word template was not available on his phone.

What is more interesting than my endless frustration, is that we showed them a few more tools we had been using, e.g. software for continuous deployment, automated testing, reverting on failure, defining infrastructure in code — in a word, “Devops”. I was surprised that none of the three of them had seen these things before. One of them was really interested, and soon after started Devops training. However, at the time (not all that long ago), it really was like showing someone an alien spacecraft. This is what reflected on them worst of all for me.

Their unreasonable demands were an emotional response to having had a prestigious project taken away from them. The ITIL framework itself, was what they had been trained to do. Not having used Devops principles, and cloud-based infrastructure was perhaps not their choice. However, I couldn’t believe that as technology professionals, in charge of one of the largest deployments of its type in the world, the fact that they seemed not to have even read about the techniques we were using, was seriously unprofessional.

Best-practice in technology changes all the time, and while some of the people I work with might learn a trick or two from my long experience, the dynamic is usually the other way around. I had been using Scrum, Terraform and CircleCI for some time at this stage, but I learned a lot about Ruby-on-Rails, and finally — after successfully faking it a few times — I internalised Agile principles properly during that year.