US Department of Defense – U.S. Air Force

The following entry is a record in the “Catalogue of Catastrophe” – a list of failed and troubled projects from around the world.

Department of Defense – USA
Project type : Integrated supply chain and logistics system
Project name : Expeditionary Combat Support System (ECSS)
Date : Nov 2012 Cost :$1B

Synopsis :
When you have 5,400 high-tech aircraft to deploy and maintain, plus 330,000 active personnel to coordinate, you have a major logistics problem on your hands! To manage those logistics the US Air Force utilizes a complex web of IT systems. Built up over many years, these systems form a patchwork of components rather than a clearly thought through and structured IT architecture. By year 2000, overlapping functions and disconnected databases meant that the Air Force was struggling to achieve the desired operational capabilities, efficiencies and financial transparency. Describing the situation, the Air Force said: “the Air Force information technology environment includes over 700 systems. Many are duplicative, stand-alone and ineffective. There is also a multitude of metrics with competing goals. Non-standardized reporting exists causing credibility issues and time- inefficiencies. In addition, there is limited visibility across the supply chain. No one knows what parts are available at different sites and personnel can’t plan for maintenance” [1].

Air force Logistics

Air Force Logistics
source U.S. Gov

To overcome the problems, the US Air Force decided to integrate their systems into a single Enterprise Resource Planning (ERP) system. As the backbone of the new system the “Expeditionary Combat Support System” (ECSS) was intended to streamline processes and bring billions of dollars in savings. Centralizing the systems through which Air Force assets are tracked, deployed, managed and maintained, the scope of the ECSS system included: advanced planning and scheduling; material management, contracting and logistics finance; configuration and bill of material; repair and maintenance; product lifecycle management; customer relationship management; order management; distribution and transportation; decision support; facilities management; quality control; document management and budgeting. Using an Oracle based Commercial-Off-the-Shelf (COTS) system the project was to adapt the code to meet the needs of the U.S. Air Force.

Originally estimates indicated that the project would take 8 years to reach full deployment and would cost $3B. Work was to be started in 2004 and was to be completed by 2012. Due to contracting disputes with the various bidders work did not begin in earnest until 2007. Quickly the project team grew and at one point with more than 1,000 team members ECSS claimed to be the world’s largest ERP project [2].

As you might expect with a project of this magnitude and complexity things did not go as planned. By 2010 signs of major problems had surfaced and between 2010 and early 2012 the project had been through no less than three project “resets”. By 2012 the Air Force had determined that the $1B spent to date had yielded negligible benefits and if they proceeded they would need $1.1B more to deploy just 25% of the original scope. Even with such a scaled back proposal, deployment would be pushed back to 2020 meaning that significant additional work and risk remained. Recognizing that a partial solution negated the overall vision of an integrated system the Air Force scrapped the Project in Nov 2012. For the $1B spent nothing could be saved. Former Presidential candidate and current Senator Sen. John McCain described the project as “one of the most egregious examples of mismanagement in recent memory”.

In a project of this size the causes of failure are often wide ranging. A report from the Institute for Defense Analysis [2] published in 2012 does however shine a light on some of the most basic structural issues with the project (and other large scale ERP implementations in other branches of the military). Among the findings:

  1. Failure to baseline existing practices and to establish effective measures for the desired outcomes,
  2. The structure of a military organization and its focus on operational capabilities rather than financial performance meant that using a COTS system that was designed for a profit making organization was a poor fit,
  3. The hierarchical decision making structures in the military were poorly aligned with the governance structure in use by the project (functional sponsors were at times lower in rank than the people whose groups were impacted by the changes),
  4. Challenges aligning and integrating functions across organizational boundaries and failure to put in place a governance structure at a senior enough level to overcome those boundaries,
  5. Lack of trust between groups,
  6. Unwillingness to adapt operating processes to match the capabilities of the software,
  7. The military’s “can-do” attitude resulted in green shifting and lack of visibility into the project’s true status
  8. Focus on schedule and budget resulted in lack of focus on actual project performance resulting in quality related issues being swept under the carpet,
  9. Seeing the project as an IT project rather than an organizational transformation project (reference tech-centric myopia)

Although the ECSS project took place in an extraordinarily complex environment and although the military’s hierarchy was clearly a factor, many of the issues identified in the report ring true with other less complex projects in the Catalogue of Catastrophe.

Update 6 Jan 2014 – A new report commissioned by the US Senate has also identified project team churn as another factor.  According to the report the project “experienced six program manager changes in eight years; five Program Executive Officers in six years; ten different organizational constructs; and the Expeditionary Combat Support System Logistics Transformation Office was staffed with term positions, not permanent positions, leading to high turnover”.

Contributing factors as reported in the press:
Failure to establish an effective governance structure. Selecting a COTS based product that was a poor fit for the project requirements. Lack of experience in large scale, complex integrated systems development and deployment. Organizational silos. Failure to effectively engage all affected stakeholders. Lack of collaboration and a lack of understanding of “change management”. Failure to establish process ownership. Project team churn.

Reference links :

  1. THE AIR FORCE’S EXPEDITIONARY COMBAT SUPPORT SYSTEM (ECSS): A Cautionary Tale on the Need for Business Process Reengineering And Complying with Acquisition Best Practices (added 6 Jan 2014)
  2. Air Force scraps massive ERP project after racking up $1B in costs