Peter Cochrane's Uncommon Sense: Formula for project failure

Why do so many software development projects crash and burn - or at least come in late and over budget? Peter Cochrane offers his explanation...

"If computers get too powerful you can organise them into a committee - that will do them in." --Bradleys Bromide

Hardly a week goes by without some press report of a massive project failure involving software in defence, healthcare, air traffic control or some other government activity. Or so it seems.

Why are there so many failures? What could be simpler than defining what you want and getting a manufacturer to deliver it?

The reality is that unlike most hardware development, software is highly non-linear and generally misunderstood. Software projects also tend to be massively over-ambitious, complex and generally suffer from a common set of failures. Broadly speaking these can be characterised as follows:

• Customers don't know exactly what they want and possess a poor understanding of software systems and the software industry.
• Vendors are competitive, extremely aggressive and out to make money.
• Specifications that are ill-defined present a moving feast of continual change during the build, deploy and commissioning periods.
• Project documentation is usually dominated by boilerplate that hides the key features and actual demands of the customer. A lot of this documentation is legalese to defend the customer's position should anything go wrong but generally turns out to be not worth the paper it's written on.
• Dispersed and poorly defined roles and responsibilities on both sides of the equation are a sure-fire way of creating one incredible mess - this is the kiss of death. More often than not this is manifest across numerous departments and management levels.
• Management arrogance allows systems to be specified, operational processes to be agreed on and maintenance routines to be defined without serious consultation with the actual users - i.e. the staff on the ground.
• Complexity seems to be a fast growing modern disease, and in software systems it is manifest in the vision that everything can be done by computer, everything can be integrated into one gigantic system and everything is possible at a price.
• Committees are the death knell of any project and large software projects in particular. I can't think of a single incidence in the history of mankind where a committee was able to create anything more powerful than a single inspirational human being who knows exactly what he is doing.

Unfortunately almost all of the above are applicable to government projects and an awful lot of industry as well. In my experience there is no one single cause of big project failure; it is always multiple causes, a selection of those mentioned above.

In order to create a successful outcome to any project it is absolutely essential that there is a champion who is ultimately responsible for it. This individual must be knowledgeable about the technology, the company employed to build the system, the actual customer requirements and most importantly the end user on the ground.

Beyond all of this, there is the customer's customer - the people who are on the receiving end of the service being provided or the systems being created. In the case of healthcare it is the patient; in the case of military system it is the enemy; in the case of retail it is the shopper.

Unfortunately it seems to be beyond the capacity of most people to keep all of these balls in the air at once. The devolution of management, the dividing up of responsibilities and the hiding of culpability behind a mire of committees, subcommittees and management hierarchy are all part of the problem.

In any mature industry it is easy to carve up a project so that each element can be defined with precision, designed, manufactured and then brought together in one assembly. For example, no one manufactures whole automobiles anymore, they are assembled from pieces - the car producers are systems integrators. They take engines from one country and put them together with a drive chain from another, a body made locally and seats imported from South East Asia.

What a car producer handles is the design capability and the jigs, fixtures and knowledge that allow the product to be realised as one entity. This is a linear system - each component can be designed and optimised individually and brought together in an optimised hole.

Unfortunately software systems do not fit the above industrial model. First of all it is immensely difficult to make software components that you can merely glue together. Second the components are not linear entities, they are inherently non linear with multiple inputs, outputs, feedback and feed-forward loops. Optimising individual components and concatenating them into a system almost always results in a suboptimum solution.

So herein lies the conundrum - how do we get from this incredibly complex world of software to the world of relative simplicity when all can be defined, broken up, parsed and brought together in one design? The reality is that for software we have some way to go - we just don't know how to do it right now. But a guiding principle at this time has to be 'keep it simple' (KIS).

Dictated on highway 280 heading south from San Francisco to Cupertino, California after a long meeting on big software projects. Despatched to my PA as a digital audio file from my Cupertino hotel via a free high speed LAN. Rewritten three weeks later and sent to silicon.com from a hotel lobby in the North of England beyond the reach of reasonable communications facilities.