Ministry of Education - New Zealand
Project type : Payroll system
Project name : Novopay
Date : Sep 2012 – May 2013 Cost : $30M NZD

Synopsis :
A 2010 report by accounting firm KPMG shone a spotlight on ineffective Project Management practices in New Zealand. Reporting that a shocking 70% of organizations included in the survey had experienced a failed project over a 12-month period, the report raises serious questions about the leadership capabilities in some of the participating organizations. This week’s entry into the Catalogue of Catastrophe is an illustration of one such failure in New Zealand.

The Novopay system was intended to be a tool that would streamline payments to the 110,000 teachers, administrators and staff throughout New Zealand’s educational system. The project had its origins in a 2005 decision that the existing payroll facilities needed modernization. Following a bid process Australia’s Talent2 was selected to both implement the new system and then operate it on a service contract basis until 2020. The original project called for the system to be implemented in 2010, but following a number of delays the project only reached operationally status in Aug of 2012.

Immediately following its operational launch problems were encountered. Some school employees reported receiving incorrect payments while others were paid nothing at all. Those problems continued to grow and the issues became headline news in New Zealand as affected employees struggled to maintain their personal finances in the face of the cash flow problems the systems failures were causing. Dubbed the Novopay debacle, at one point affected staff had reported more than 18,000 payroll errors and the operational staff supporting the system appear to have been overwhelmed by the amount of manual intervention needed to correct those errors.

Tracking and analysis of the errors identified after the launch, identified more than 500 distinct defects in the system. Of those 44 were deemed to be very serious. In Aug of 2012 when the system went live reports indicate that only 147 defects were known meaning that Quality Assurance testing had failed to identify several hundred problems in the system. Many of those problems were traced back to errors in the original project requirements and the design of the new system that allowed incorrect data to be entered into the system thereby leading to incorrect payroll payments and related problems.

A Mar 2013 review performed by Deloitte raised serious questions about the stability of the system and outlined a 1-year remedial plan that needed to be followed to ensure the operational stability of the system and that the originally planned business benefits were realized.

Contributing factors as reported in the press:
Requirements problems and poor system design (a – usability issues, b – failure to establish appropriately robust data validation steps to ensure the quality of data input into the system and c – poorly designed reports that inhibited management oversight of the system’s use). Failure of quality control tests (failure to identify data corruption issues and logic flaws). Implementing the system with a high number of unresolved defects. Lack of control over manual intervention processes used when problems were encountered.

See also (other recent failed payroll projects):

1. Marine country
2. MyCalPAYS
3. Los Angeles Unified School District

Reference links :

1. The Novopay website
2. The payroll system that doesn’t pay
3. Charting the Novopay debacle
4. Ministry of Education Novopay Technical Review
5. Novopay errors rise

J.P. Morgan Chase  & Co. 
Project type: Financial risk analysis tool
Project name: New Synthetic Credit VaR (Value at Risk) Model
Date: Sep 2011 (project) – Apr-Jun 2012 (operational failure)  Cost: Approximately $6B

Synopsis:
Sometimes the mightiest of the mighty is humbled by the meekest of the meek. Microsoft Excel may not be the most grandiose software tool in the market, but it’s amazing capabilities mean that it is one of the most widely used there is. As those of us who are regularly users know, there is however a dark side to the mathematical marvel that Excel has become. As you are absorbed into the wizardly magic of its number crunching capabilities, it is all too easy to make a mistake, and, once your formulas are wrong, it be can be very hard to see that you have gone wrong.

J.P. Morgan Chase, one of the world’s most mighty banking and financial services firms, is one organization that has learned the risks of Excel the hard way. In an incident that drew worldwide attention, J.P. Morgan lost billions of dollars in the so called ”London Whale” incident. The London Whale was a trader based in J.P. Morgan’s London Chief Investment Office (CIO). He had earned his nickname because of the magnitude of the trading bets he was making. It is said that his bets were so large his actions alone could move a market. Despite his undeniable power, things went seriously wrong between Apr and Jun 2012 and a poorly positioned trade resulted in losses that eventually totalled up into the billions of dollars.

According to available reports, the part of the CIO office involved was responsible for managing the bank’s financial risk using complex financial hedging strategies in the derivatives markets. To support the operations J.P. Morgan had developed a “Synthetic Credit Value at Risk (VaR) Model” that helped them understand the level of risk they were exposed to and hence make decisions about what trades they should be making and when.

The tool had been developed in-house in 2011 and was built using a series of Excel spreadsheets. According to J.P. Morgan’s own report to their shareholders that was published following the disaster, the spreadsheets “had to be completed manually, by a process of copying and pasting data from one spreadsheet to another”. To pick what appears to be an appropriate word in this particular case: YIKES! Immediately any serious user of Excel would know that relying on copy and paste is risky business. One minor slip and the data you have isn’t what you thought it was. One accidental move and you can wipe out the embedded formulas without realizing what you’ve done. Relying on copy and paste in a tool that supported billion dollar transactions seems unfathomable to me.

J.P. Morgan’s internal report into the incident illustrates how failures in the project that developed the tool were the driving forces that lead to the debacle. The report is interesting reading and illustrates that despite the financial risks involved a cavalier approach to developing the tool was used. According to the report six issues were contributing factors that left the London Whale with a tool that gave him the wrong advise. Quoting from the report, the investigating committee found:

1. “Inadequate resources were dedicated to the development of the model. The individual who was responsible for the model’s development had not previously developed or implemented a VaR model, and was also not provided sufficient support – which he had requested – in developing the model.
2. The model review policy and process for reviewing the new VaR model inappropriately presumed the existence of a robust operational and risk infrastructure similar to that generally found in the Firm’s client-facing businesses. It thus did not require the Model Review Group or any other Firm unit to test and monitor the approved model’s implementation. Back-testing was left to the discretion of the Model Review Group before approval and was not required by Firm policy. In this case, the Model Review Group required only limited back-testing of the new model, and it insufficiently analyzed the results that were submitted.
3. The Model Review Group’s review of the new model was not as rigorous as it should have been and focused primarily on methodology and CIO-submitted test results. The Model Review Group did not compare the results under the existing Basel I model to the results being generated under the new model. Rather, it theorized that any comparison of  the numbers being produced under the two models was unnecessary because the new model was more sophisticated and hence was expected to produce a more accurate VaR.
4. The model was approved despite observed operational problems. The Model Review Group noted that the VaR computation was being done on spreadsheets using a manual process and it was therefore “error prone” and “not easily scalable.” Although the Model Review Group included an action plan requiring CIO to upgrade its infrastructure to enable the VaR calculation to be automated contemporaneously with the model’s approval, the Model Review Group had no basis for concluding that the contemplated automation would be possible on such a timetable. Moreover, neither the Model Review Group nor CIO Risk followed up to determine whether the automation had in fact taken place.
5. The CIO Risk Management played too passive a role in the model’s development, approval, implementation and monitoring. CIO Risk Management personnel viewed themselves more as consumers of the model than as responsible in part for its development and operation.
6. The CIO’s implementation of the model was flawed. CIO relied on the model creator, who reported to the front office, to operate the model. Data were uploaded manually without sufficient quality control. Spreadsheet-based calculations were conducted with insufficient controls and frequent formula and code changes were made. Inadequate information technology resources were devoted to the process. Contrary to the action plan contained in the model approval, the process was never automated.”

As has happened before in other organizations, a lack of due diligence and cavalier management oversight allowed the humble Excel spreadsheet to once again claim another victim. What makes this story so important is that the London Whale incident is likely to be the largest loss that can be directly attributed to the use of Excel.

Contributing factors as reported in the press and the court proceedings:
Lack of risk management in deciding how best to implement the new tool. Lack of quality control. Poor tool selection decision. Failure to follow through with quality related action items identified in a risk assessment review. Project governance and oversight failures. Failure to assign appropriately skilled resources to the project (despite the request from the team).

Notes: J.P. Morgan is not the only organization to be caught out by Excel.

1. In last week’s post we outlined the BSkyB versus EDS court case. Again an error in a spreadsheet was partly to blame for pricing errors that in part lead to the £700M legal wrangle between the two parties.
2. Another well publicized case is the “Growth in a Time of Debt” story in which two world leading academics (Reinhart and Rogoff) wrote an academic paper analyzing the effect of national debt on economic growth. Published in 2010 the paper’s findings were used by some politicians to argue that austerity policies needed to be adopted to manage national debt despite the fact that cutting government spending could further compound the 2008 Great Recession. The resulting austerity steps taken by countries around the world caused massive pain and some feel that the polices did indeed deepen the global slump. In 2013 a Ph.D. student at the University of Massachusetts (Thomas Herndon), tried to replicate Reinhart and Rogoff’s results. In doing so Herndon got hold of the original spreadsheet used by Reinhart and Rogoff and discovered that there were errors in the formulas they had used. Some argue that those errors lessened the effects reported in the paper and hence undercut the austerity argument made by the politicians. News of Herndon’s findings spread throughout the globe and Reinhart and Rogoff accepted the mistake by publishing an “errata” to correct the error. The argument however continues as to whether or not the politician’s interpretation of the paper was correct and whether the errors affected their argument substantively. If the argument were wrong and nations underwent their austerity programs on a false foundation, the impact may have been magnitudes bigger than the losses of the London Whale.

Robert Goatham – Editor

Reference links:

1. Report of JPMorgan Chase & Co. Management Task Force Regarding 2012 CIO Losses
2. Microsoft’s Excel Might Be The Most Dangerous Software On The Planet
3. How the Case for Austerity Has Crumbled

British Sky Broadcasting (BSkyB) Limited – Middlesex, UK
Project type: Customer Relations Management system
Project name: Sky CRM Project
Date: Jan 2010 court case concluded   Cost: £265M

Synopsis:

Call center worker
(source wiki commons)

2010 marked the end of another chapter in what is reputed to be IT’s longest and most expensive court battle: the battle between BSkyB (a digital television service provider, telecom and media organization based in the United Kingdom) and Electronic Data Systems Corp. (EDS). The case has its roots in BSkyB’s Feb 2000 announcement that they were planning on upgrading their customer relationship management software. As a successful organization in a growing market, BSkyB had found that their existing Digital Customer Management System (DCMS) was unable to keep up with the service levels customers were expecting and as a result their “churn rate” (the rate at which customers were leaving BSkyB) was too high. To overcome the problem BSkyB initiated the “Sky CRM Project”, with the mandate of sourcing for and implementing a new enterprise wide Customer Relationship Management (CRM) System.

To initiate the process, BSkyB turned to Arthur Andersen (AA) consulting to review their current CRM practices and to map out a strategy for moving forward. In an initial review presented in Jan 2000 AA suggested that implementing the new systems would cost anywhere between £60M and £90M depending on the assumptions used. They also proposed that the project could complete an initial implementation (in a single newly built call centre) within 9 months of project initiation (assuming the call centre construction work were already complete) and that conversion of BSkyB’s other existing call centres would be challenging but could be done in a further 6 to 9 months.

Following initial internal planning an Invitation to Tender (ITT) was published on 17 Mar 2000.  The document indicated that the project objective was: ”The Build and Implementation of a World Class Contact Centre for BSkyB and the further retrospective fitting of environment, culture, process and technology to existing sites” and that the target for the first implementation would be around the 9 month mark. The intention was to equip a new contact centre (in Manchester) that would first utilize the new systems (phase 1) and then to retrofit the new system into the existing call centres in Dunfermline and Livingston (phase 2).

Three primary bids were received. One by PricewaterhouseCoopers (PwC), one by EDS and one from AA (although AA subsequently withdrew and partnered with EDS as a sub-contractor on the EDS bid). Following bidder presentations, EDS was selected because BSkyB was impressed by EDS’s assertion that the work could be completed in just 18 months. Under the proposal EDS was to be the systems integrator and they were to oversee the implementation of the CRM software supplied by Chordiant Software Inc. To allow work to proceed a Letter of Intent was signed on the 9th Aug 2000 indicating that the first phase would be implemented by 1st April 2001.

Unfortunately, the relationship between BSkyB and EDS got off to a shaky start. Even before the contract had been signed EDS advised that the delivery dates and budgets they had committed to in their original tender could not be met. As a result BSkyB dropped the Manchester call centre from the project and redefined the project phases. Despite that initial hick-up the prime contract was signed on 30th Nov 2000 and the Sky CRM Project was officially launched with a budget of £50M and a new phase one “go-live” delivery date of 31st July 2001.

As with many large scale IT projects, the project did not proceed as planned. The phase one delivery schedule slipped again and BSkyB and EDS agreed to new delivery dates in a Letter of Agreement dated 16th Jul 2001. Despite the extensions the project continued to have difficulties and although phase 1 did go live on 19 Oct 2001 the system caused significant operational issues. Reports indicate that following live release there were 718 “severity 1″ issues reported.  Typically in such projects a “severity 1″ issue is an issue that has system wide implications and /or the potential to cause significant impact to customers. This painful deployment period lasted until Feb 2002 and only at that point was a manageable degree of operational stability reached.

Following the problems BSkyB took things back into their own hands and in Mar 2002 signed a new Memorandum of Understanding (MOU) with EDS that redefined the EDS role and the scope of work. Notably, the MOU meant that BSkyB would assume the systems integrator role and that Phase 2 would be split into five stages 2.1 through 2.5. EDS would however continue to supply staff to the project.

Following the project “reset”, stage 2.1 was implemented on 14th Nov 2002, stage 2.2 on 21st Mar 2003 and a de-scoped version of phase 2.3 finished its live implementation in Mar 2006. After that the remaining scope was dropped and the project terminated. At the time of closure the project had cost a whopping £265M instead of the original £50M budgeted.

In December 2003 BSkyB initiated legal proceedings against EDS. In a claim that would eventually total more than £700M BSkyB sued for “deceit, negligent misrepresentation and breach of contract”. At the heart of the issue was the allegation that EDS had not done sufficient due diligence in submitting their bid and had as a result had knowingly given commitments to BSkyB that were not based upon a solid project plan. Those commitments implied to BSkyB that EDS had sufficient qualified staff to complete the project, that those staff were ready to start work immediately, that the technology to be used was proven, and that the work could be completed in the required timeline and at the stipulated cost. Those commitments became the foundation upon which BSkyB’s expectations of the contract were based and the platform upon which BSkyB selected EDS as the winning bid.

As the sequence of events that followed the acceptance of the EDS bid demonstrated there was a gap between BSkyB’s expectations and what actually transpired. Resourcing issues, project delays and quality problems were reported as the project progressed and as the court documents show the representations made during the bid process were not accurate predictors of what would eventually happen.

Many of the issues were related to the way in which the EDS bid was put together and the representations EDS made to BSkyB through the bidding process. The bid was seen by BSkyB as a firm commitment, but the reality of the situation was that there was no firm scope baseline upon which to base a firm commitment. Errors were made in calculating the price (the court case reports Excel calculation errors which resulted in under bidding on price), contingency was stripped out of the contract (presumably to make the bid more attractive to BSkyB) and there was no detailed resourcing plan for the project at the time the bid was made. Court proceedings further revealed that an internal “red team” review performed by EDS in Aug 2000, reported that the scope lacked definition and shortage of staff (quantity and skills sets) were problems on the project.  The red team noted that they did not believe the system could be implemented by the then target date of 1 Apr 2001.

Following a lengthy legal process, in January 2010, BSkyB was awarded a settlement of £200M. The Honourable Mr. Justice Ramsey of the England and Wales High Court (Technology and Construction Court) determined that EDS had breached the contract and performed misrepresentation. He added that without a new CRM system, BSkyB suffered business loss in terms of “churn”.

In his lengthy decision, the Honourable Mr. Justice Ramsey detailed significant faults, mostly by EDS – but not all. Although the judge did not rule in favour of all of BSkyB’s allegations, the ruling did establish that EDS had not done sufficient planning before giving assurances to BSkyB that they could deliver the project according to the agreed schedule and budget. As such EDS had misrepresented what they would be able to deliver. He also found that had the truth been disclosed to BSkyB they would have chosen the PwC bid and hence taken a very different path.

Note: EDS countered with a claim of £4.8M for unpaid invoices and so the story has not quite reached its final end just yet.

The editorial team

Contributing factors as reported in the press and the court proceedings:
Failure to define scope before committing to schedule and budget to a customer. Failure to communicate the risk of the lack of scope definition to the customer during the bid process and through the contractual arrangements. Ineffective resource planning. Resource skill set shortfalls in some of the staff assigned. Poor program management. Failure to document the project effectively. Poor quality management practices resulting in the problems with the phase 1 deployment (e.g. the test environment used was not an accurate replication of the production environment in which the system was to function).

See also:

1. J.P. Morgan – Excel spreadsheet error

Reference links:

1. BSkyB Selects Chordiant Software for Multi-Million Dollar Unified Customer Relationship Management Strategy – Original press release from Chordiant
2. Media Giant BSkyB sues EDS over troubled CRM system - by Marc L. Songini, dated August 17, 2004
3. BSkyB and EDS face £45M legal bill over CRM system –  Tony Collins, dated October 29, 2007
4. EDS BSkyB dispute became IT’s longest and most expensive court battle
5. England and Wales High Court (Technology and Construction Court) Decisions –  dated January 26, 2010
6. BSkyB v EDS judgment at long last – a dodgy degree, a dog called Lulu and some lessons for both customers and suppliers - by Mills & Reeve, dated February 2010

Many of the better Project Management courses in the market talk about how we need to be sensitive to the impacts a project will have on people and the need for “Change Management”, but often there is no framework put forth to help people think about how a project might impact the stakeholders. With that in mind, I’ve been thinking about the dimensions within which a project could impact its stakeholders.

Based on observation of the project’s I’ve been involved in, and analysis of the projects in the catalogue, there are common themes that offer up the hope of developing a more comprehensive framework for analyzing the impact a project may have. The goal of any such framework would be to help organizations understand the potential sources of resistance to change in their projects so that they are better able to engage with their stakeholders.

At the most basic level resistance to change can occur whenever the status quo is challenged or disrupted. The status quo is the baseline against which change takes place and many people feel discomfort or anguish when pushed too far out of the “comfort zone” that is created by familiarity. To understand where resistance to change may come from I think we need to understand the dimensions that could be considered part of the status quo.

Obviously different things are important to different people, but observation of people’s behaviours in real life projects illustrate how there are often common elements, that if changed, can be a source of problems. Dimensions that may be considered as being a part of the status quo might include elements such as:

1. Familiar living or working conditions and environment
2. Our sense of belonging to something (being part of a team, a work unit, an organization, a family, etc)
3. Our sense of security (financial, job, relationships, etc) arising from the status quo
4. Our sense of self worth arising from our part in the world around us or the contributions we feel we are making

If a project changes anything related to those dimensions we have a potential for resistance. If the changes are clearly beneficial to the stakeholder, and they see and accept those benefits, then there is unlikely to be resistance to change. If however there are no benefits for the stakeholder or they don’t see the benefits for some reason, resistance can occur. To make those dimensions a little more tangible, here is a set of questions that can help deepen the level of stakeholder analysis in a project:

1. Does the project have any structural impacts?
   1. Changes in a person’s level of autonomy in the workplace
   2. Changes in power or authority (e.g. a change that results in someone being in a different level in the hierarchy)
   3. Changes in working relationships (e.g. breaking up people who have worked together for a long time)
   4. Changes to job security
2. Does the project have any environmental impacts?
   1. Changes in working or living conditions
   2. Changes in the workplace that will have some form of impact on a person’s personal life (e.g. some form of change that affects work/life balance).
3. Does the project impact anyone financially?
   1. Changes in income
   2. Changes in the way in which people earn or are paid
   3. Changes in the relative income of one group or individual versus another
4. Does the project impact they people perform their work?
   1. Changes in processes used to perform work functions
   2. New tools
   3. Changes in workload

Stakeholder analysis is a weak spot in many projects and in part it is because people are not thinking about the different dimensions in which a project may impact someone. In large part the problem arises from the training people receiving. Most Project Management training says you need to analyze stakeholders, but then stops short of identifying the dimensions you should be considering. Hopefully the above list can act as a catalyst that gets people thinking about stakeholder analysis at a deeper level.

Robert Goatham – Editor

Airbus SAS - France
Project name : A380
Project type : Commercial aircraft development
Date : Dec 2000 – Oct 2007
Cost : $6.1B in additional costs due to project delays^*1

The Airbus A380 (source wiki commons)

Synopsis :
I’ve often written about decision-making in the project environment and how the decisions made in a project shape the outcomes the project achieves. Often when a project gets into trouble it is the result of the interaction of many poorly made decisions. Occasionally it is a single decision that can be isolated out as the source of trouble. The Airbus A380 is one such project.

As the world’s largest commercial aircraft the Airbus A380 is a feat of engineering. With two full decks, a wingspan wider than a football pitch and space for up to 850 passengers (in high density mode), the A380 is the most complex aircraft flying today. While the aircraft has now been in operational service for 6 years, the project that created this behemoth suffered its fair share of problems and delays. Originally scheduled for delivery in 2006, the aircraft’s entry into service was delayed by almost 2 years and the project was several billion dollars over budget.

At the heart of the problems were difficulties integrating the complex wiring system needed to operate the aircraft with the metal airframe through which the wiring needed to thread. With 530Km of wires, cables and wiring harnesses weave their way throughout the airframe.  With more than 100,000 wires and 40,300 connectors performing 1,150 separate functions, the Airbus A380 has the most complex electrical system Airbus had ever designed. As the first prototype (registration F-WWOW) was being built in Toulouse France, engineers began to realize that they had a problem. Wires and their harnesses had been manufactured to specification, but during installation the wires turned out to be too short. Even though the cables were at times just a few centimetres too short, in an aircraft you can’t simply tug a cable to make it fit.  As construction of the prototype continued Airbus management slowly came to the realization that the issue was not an isolated problem and that short wires was a pervasive issue throughout the design.

Internal reviews identified that the heart of the problem was the fact that the different design groups working on the project had used different Computer Aided Design (CAD) software to create the engineering drawings. The development of the aircraft was a collaboration between 16 sites spread across 4 different countries. German and Spanish designers had used one version of the software (CATIA version 4), while British and French teams had upgraded to version 5. In theory, the fact that the design centers were sharing their drawings meant that the electrical system designed in Germany would be compatible with the airframe components designed in France. Part way through the project the design centres also started integrating their diagrams into a single 3D Digital Mock-up Unit (DMU) that should further have validated compatibility. Unfortunately, the construction of F-WWOW demonstrated that theory and practice are not always the same thing.

In part the problem was the CATIA version 5 was not a simple evolution from version 4, it was a complete rewrite. Reports indicate that the calculations used to establish bend radii for wires as they wove through the airframe were inconsistent across the different versions of the software and that inconsistency resulted in the problem. Stripping out the wiring from the prototype, redesigning the wiring, making new harnesses and then rethreading the wiring into the airframe became a monumental task. Taking months to complete the project was delayed multiple times as hundreds of engineers tried to overcome the problems. At one point more than 1,100 German engineers were camped out at the Toulouse production facility trying to rectify the problems.

The root of the problem can be traced back to a single decision: the decision to proceed with the project despite the fact that two CAD systems were in use. That decision resulted in design inconsistencies, mismatched calculations and configuration management failures.

As for many failed decisions there is a lot of context that lead up to that decision. Much of that context had to do with the history of Airbus. Prior to the 1960′s each European power had its own aerospace industry. As aircraft development projects became larger, more complex and more risky, these separate entities found it hard to compete. To overcome the problem European governments began cooperating with each to integrate their individual aerospace expertise into an organization that was large enough to compete with the US based Boeing Commercial Aircraft company. Originally set up as a consortium of separate organizations, Airbus’s first aircraft (the A300) became operational in 1974. Several highly successful aircraft designs followed and by the late 1980′s the consortium was able to compete directly with Boeing. Building on those successes Airbus took the next step and became a single corporate entity in 2001.

As those who have done merger type projects know, merging disparate entities into a single homogeneous whole is not easy. Because of their differing origins, different parts of the organization inherit different corporate cultures, management styles and IT systems. Those differences can be both hard and expensive to overcome and at Airbus a number of such differences were still deeply entrenched when the A380 project began. Even at the very top of the organization there was an elaborate split between French and German control (co-CEO’s) and in 2001, when the A380 Program Manager attempted to move the German designers onto the same CAD system as the French, he met a wall of resistance. Personal rivalries and national pride^*2 are reported to have been issues that stood in the way and ultimately the pressure to keep the project moving forward meant that the CATIA version issue was never resolved. Five years later that issue was to cost him his job and he resigned as part of the house cleaning that resulted once the magnitude of the issue became clear.

Eventually F-WWOW did fly (first flight video) and the aircraft is indeed a wow. However, that one small decision point became the seed from which a billion dollar delay matured. Of course, interoperability of design tools is not only an Airbus issue. In today’s complex integrated supply chains, stories of failed configuration management during the design process are a sadly all too common.

Contributing factors as reported in the press:
Configuration management failure. Overly complex organizational structure that attempted to keep different parts of the organization happy rather than focusing on how best to build the aircraft. Failure to form a single project team across the multiple design centres in use. Overly aggressive schedule leading to schedule pressure such that key issues were ignored early in the project lifecycle. Failure to address issues when they were first identified resulted in snowballing costs and significantly higher costs once the problems were finally faced up to.

Related stories:

1. Boeing 787 Dreamliner

Reference links:

1. BBC – Q&A: A380 delays
2. The Airbus saga: Crossed wires and a multibillion-euro delay
3. Airbus Vows Computers Will Speak Same Language After A380 Delay
4. What Grounded the Airbus A380?
5. Airbus: First, Blame the Software
6. A380 Program Update – Global Investor Forum – Oct 2006 (see slide 13 onwards)
7. EADS execs pledge to restore confidence in Airbus, fix A380 problems

The following post is a “Lesson Learned” that comes from the analysis of the failed projects documented in the “Catalogue of Catastrophe” or from the experiences the editorial team have had working with clients around the world. The post is published here to spark discussion and help organizations think about what it takes to improve project success rates.

In today’s competitive markets sellers sometimes feel the pressure to use aggressive marketing tactics in order to sell their products or services. As buyers of professional services know, one such marketing tactic is “bait and switch”. In a bait and switch, the seller uses their most seasoned professionals to do the sales pitch, but once the contract is signed, the seller assigns much more junior staff to actually perform the work.

In legal terms “bait and switch” is defined as being a marketing tactic whereby a vendor advertises one product or service of a certain price and quality, yet when the customer wishes to order that particular product, they are advised that the item is no longer available, and they are pressured into considering a more expensive and/or lesser quality substitute. Bait and switch has its origins in the retail sector, where laws protecting the consumer are well established. Unfortunately, in the professional services industry bait and switch is also being used and the legal situation is much less clear. Marin County (Marin) vs. Deloitte Consulting LLP (Deloitte) (MERIT Project) is a recent example¹ in which a buyer accused a seller of using a bait and switch type tactic

In the Marin vs. Deloitte case, Marin alleged that Deloitte did not provide Marin with the quality team that Marin expected from the presales presentation. Following the failure of the project, Marin decided to take the issue to the courts. Following the legal process the Court disagreed with Marin’s view of the situation saying that any assertions made during the bid process were just “puffery” (statements that were obviously exaggerated and that the average customer would not fall victim to such behavior ²).

Given that this issue has been around for such a long time and many buyers have seen the issue first hand, it’s time that buyers matured a little and took reasonable steps to protect their interests. Although there is no silver bullet that assures avoidance of bait and switch, when contracting professional services, there certainly are steps you can take to reduce the risk.

1. Do not rely on pre-contract representations as being an assurance that key vendor staff will be allocated to your project - Unless there are facts regarding personnel achievements and expertise that are identified in the contract, you cannot expect to benefit from them.
2. Do not give the responsibility of monitor and control of personnel movement to the vendor - You must be in charge of approval and acceptance of key personnel movement on and off the project.
3. When evaluating vendor staff, do not rely on resumes - Resumes are personnel marketing tools and hence the potential for “puffery” is high. Instead consider putting in place a standard fill in template (a “skills and experience disclosure statement”) that staff assigned to the project by the seller are required to complete and submit to you for approval before the staff are accepted into the project. The disclosure statement template should be structured to elicit straightforward employment and experience facts (years of experience – size of projects worked on – roles performed – number of people directly supervised, etc) rather than superlatives or other subjective language.
4. Require an Org Chart - The vendor must provide an organizational chart that defines the key team players. The org chart will also serve as a basis for understanding team member responsibility, making it a particularly useful tool when faced with the impact of personnel replacement.
5. Do not assume that the initial team will remain on your project - People have to leave projects for some very legitimate reasons. You must ensure that they are replaced by people of at least equal qualifications. The contract must include a process for key personnel replacement i.e. how key people are to be replaced and how the replacements are approved.

Resolution strategies to consider:

Developing a skills and experience disclosure statement for inclusion in your RFP and contract terms /Relying on project personnel proven skills and knowledge only / Requiring an organizational chart in responses to your RFP / controlling key personnel movement on and off your project team.

Contributing Editor: Brig Henry

References:

1. Superior Court of California, County of Marin – May 28, 2010
2. Deloitte’s Puffery in their RFP to Marin County and What it Means for Current and Future Clients

The following post is a “Lesson Learned” that comes from the analysis of the failed projects documented in the “Catalogue of Catastrophe” or from the experiences the editorial team have had working with clients around the world. The post is published here to spark discussion and help individuals and organizations think about what it takes to improve project success rates.

Given the rapid pace of change in most markets and the need to use resources as efficiently as possible, organizations want things done as fast as possible. Throughput is the mantra: get as many projects completed in as short a period of time as you possibly can. Meet the customers needs, keep up with competition, pursue the latest ideas, JUMP when the C-suite says JUMP!

While such pressures are fully understandable, organizations need to be careful that they don’t bite off more than they can chew. Like the proverbial kid in the candy store, organizations can easily end up with more projects in progress than they have the capability to either manage or deliver. When that happens there can be serious side affects. Staff can become overloaded, stress levels can build and productivity can drop. Priorities keep shifting, confusion builds and staff loose faith that their senior leadership team have the skills to successfully move the organization forward. Falling morale and a burned out workforce slow the pipeline of work still further and the organization runs the risk of entering a death spiral in which many projects get initiated, but far fewer get finished. Given that business results come from projects delivered, not concepts floated or work in progress, biting off more than you can chew can be the slipper slope down which a business failure builds up its momentum.

Of course one way to improve the situation is to get control over the portfolio of projects and match the flow of projects to the available resources (project portfolio management combined with enterprise resource management). Those two are basics that every organization needs to put in place. Beyond that the burning question is: how do we increase project productivity as much as we possibly can?

Working as both a consultant and a trainer provides me the opportunity to work with people from many, many organizations. As such I’m able to contrast successful organizations with organizations that are struggling. In doing so there are some clear lessons learned that can help organizations become as productive as they can be. For discussion purposes here is a sampling of some of the most important themes that come through from that contrast;

1. Prioritize skills & training over adding layers of bureaucracy - Projects move quickly when people with the right skills and knowledge are able to make decisions without being encumbered by unnecessary bureaucracy. Some organizations continually under invest in skills development and skills acquisition and instead try and rely on levels of bureaucracy to ensure good decisions get made. Skills are the multiplier of productivity while layers of unnecessary bureaucracy are the brakes that bind on the wheels of progress.
2. Physically co-locate the team - Where teams are directly in touch with each other communications flows can dramatically improve. Relationships can build and decisions can be made much more swiftly. While “virtual-team” technologies are good and getting better, there is still no substitute for physically sitting next to someone on a daily basis. Where cross-disciplinary teams are able to co-locate things can really start to happen.
3. Dedicate the team to the project - Where team members are supporting many projects simultaneously they are juggling multiple balls. Each time they shift between projects they are context shifting and loosing focus. Successful organizations tend to be the ones who allow their people to focus.
4. Stabilize commitments and key decisions - The role of the Project Sponsor is in many ways the most important, where sponsors fail to understand their role or keep flip-flopping their decisions the foundation upon which the project stands is in continual flux. Sponsors need to think hard about what it is they want so that projects get off to a clean start. Poorly considered project initiations lead to churn and inefficiencies that sap energy from the organization.
5. Establish a meaningful process - While skills matter, so too does having an efficient process. A good project lifecycle ensures the right decisions are made in the right sequence and at the right time. A lack of process means decisions are likely to churn and that chaos will burn.
6. Communicate frequently and meaningfully - Shorter, but more rapid and efficient interactions are generally better than long more spread out interactions. Inefficient meetings and poor communications are one of the biggest problems some organizations face. Meetings and communications are tools to be used, sadly few organizations train their staff in how to use those tools effectively.

Periodically organizations need to slow down and consider whether their practices are supporting or hindering productivity. A facilitated session where the management team steps backs and takes a hard look at their practices can be a great thing to do. The above list is food for thought, but often the best ideas come from within. Failure to do so often leaves the organization surrounded by fires. When you’re surrounded by fires you may end up simply jumping up and down rather than taking that all important leap forward.

Resolution strategies to consider:

Investing in training (both for project team members and sponsors) / co-location of the project team / project portfolio practices / enterprise level resource management / empowered team members / streamlined processes and methodologies / bureaucracy reduced to the minimum necessary.

Robert Goatham - Editor the Why Projects Fail blog

Marin County - California, USA
Project type : Enterprise Resource Planning system – Accounting package
Project name : MERIT (Marin Enterprise Resource Integrated Technology)
Date : Jan 2013 court case settled  Cost :$33M

Synopsis :
Marin County (Marin) is an affluent county of about 250,000 inhabitants located just across the Golden Gate Bridge from San Francisco. In 2003, Marin commissioned a study to assess the effectiveness of its financial management, human resources (HR) and payroll systems. The study determined that Marin’s “existing [IT] systems were outdated and inconsistent with industry standards [and]…..were largely dependent on manual, time-consuming processes”.

As a result, in 2004 Marin initiated the Marin Enterprise Resource Integrated Technology (MERIT) project with the goal of replacing the affected legacy systems. Issuing a Request for Proposal (RFP), Marin selected Deloitte Consulting LLP (Deloitte) to implement the MERIT system and accepted Deloitte’s assurances that they had the necessary skills to complete the project successfully. Thus in March 2005, Marin signed a contract with the well-known software consulting firm to implement SAP’s Public Sector ERP system.

Release I of the new system went live on July 3, 2006. Almost immediately, the system experienced “significant cash reconciliation and financial posting issues”. On January 3, 2007, despite the fact that Release I was still experiencing problems, Release II went live as well. The second release introduced further problems and Marin found themselves struggling with performance problems and other very significant flaws. Shortly thereafter, Marin fired Deloitte and initiated a “get well” program in an attempt to recover the situation.

Following internal reviews of the situation, Marin sued Deloitte in May 2010. The case alleged fraud and sought $30M in damages. The key allegations were that Deloitte:

1. Misrepresented its skills and experience during the RFP processes and after contract signing failed to resource the project with staff who had the necessary skills to complete the work successfully (the so called “bait and switch” manoeuvre)
2. Withheld critical project risk information
3. Committed conflict of interest by influencing Marin’s own Project Manager into accepting incomplete work and concealing project risks from senior management so that the two software releases could go live.

Over the next year, Marin expanded its legal action to include its then former Project Manager and SAP themselves. The case came to a conclusion in Jan 2013 with the Court finding that the only legal issue was scope of work, and that SAP and Marin’s former Project Manager had not engaged in any wrong-doing. Marin and Deloitte finally settled January 9, 2013. The Settlement amounted to Deloitte refunding Marin $3.9M of their $11M system integration fee. The legal fees for reaching the settlement are report to have cost Marin a total of $4.7M, while the failed project itself is reported to have cost the Marin taxpayers an estimated $28.6M.

The MERIT project raises three important issues that are commonly seen in projects:

1. The alleged use of “Bait and Switch” tactics by sellers. On this issue Marin claimed that they did not get the high level of talent and expertise they had been promised during the RFP process (in fact once the project was underway Marin claimed that some of the Deloitte staff had to attend the same basic SAP classes as the Marin staff). In its judgment the Court disagreed with Marin’s claim, saying that Deloitte’s actions during the RFP processes was one of “puffery” (i.e. “promotional statements and claims that express subjective rather than objective views and which no reasonable person would take literally” – Newcal Industries, Inc. v. IKON Office Solutions, 513 F.3d 1038, 1053 (9th Cir. 2008) – Wikipedia). Suffice to say buyers should be careful to not fall victim to marketing hype, and try to protect themselves from any Bait and Switch type tactics.
2. The failure to define adequate acceptance criteria when planning and in accepting the live release of both Release I and Release II. Given that both Releases were failures, clearly Marin’s acceptance criteria were inadequate, and not controlled by their own team.
3. Lack of quality control and inadequate project governance by Marin. The failure to oversee the work of the Project Manager and establish an appropriate robust acceptance processes appear to have been direct causes of the failure.

Contributing Editor: Brig Henry

Contributing factors as reported in the press:
Insufficient skilled people acting for the best interests of the buyer. Reliance on pre-contract hype and falling for aggressive marketing tactics by the seller. Unclear scope of work assigned to contractor. Different opinions about what subjective words meant (what does it mean when the contract says a person shall have “significant experience”). Failure to test the system effectively. Failure to establish an effective sign-off and acceptance process.

Reference links :

1. Marin County cuts its losses, settles computer lawsuit for $3.9 million
2. Original contract
3. Court filing - Superior Court of California, County of Marin May 28, 2010
4. County of Marin v. Deloitte Consulting LLP et al – Document 123
5. Lessons learned, supervisors prepare to replace costly, troubled computer system
6. Deloitte’s Puffery in their RFP to Marin County and What it Means for Current and Future Clients

State of California - California, USA
Project type : Payroll and benefits system
Project name : 21st Century Project (MyCalPAYS) 
Date : Feb 2013  Cost :$254M

Synopsis :
As with many large organizations, the state of California has over the years built up a complex interconnected set of IT systems to support their daily operations. Managing multiple systems can however be costly. There is often duplicated data spread across different databases, complex business processes to help bridge between systems and elevated maintenance and training costs. As has been the trend for the past 15 years, the State of California decided to integrate their systems into a single Enterprise Resource Planning (ERP) solution. Merging 13 separate systems into a single system, the 21st Century Project was to improve efficiency and reduce the cost of handling payroll and benefit payments for the state’s 243,000 employees.

The origins of the 21st Century Project date back to 2004 when $70 million was spent in an initial attempt to integrate payroll functions. The contractor responsible for that initial attempt was terminated in 2009 due to a lack of progress and a fresh attempt was made under the title MyCalPAYS and with a new plan to use an SAP based system. Said to be the largest payroll project in the US in 2012, the MyCalPAYS system was deployed in a phase 1 release in June 2012. Deployed to a single organizational unit with just 1,300 staff (managed under 2 union agreements), the system immediately ran into problems. Staff were being paid incorrectly, simple math functions are reported to have been inoperable and incorrect allowances were being handed out. An announcement from the State Controller’s Office noted that “The errors in the SAP system affect everyday lives. Not only have SCO employees been paid too much, or too little, they and their family members also have been denied medical services despite paying for the insurance coverage. Payments to the State’s dental, vision and deferred compensation partners have been incorrect and delivered late. Improper deductions have been taken, payments have been made to the wrong payee, payroll and pensionable wages have been incorrectly calculated, and union deductions incorrectly determined.”

State officials had to undertake significant investigation and recalculations to correct the multiple problems the system was producing and after 8 monthly cycles, the system had failed to produce a payroll that didn’t contain significant deficiencies. Reports indicate that hundreds of separate problems were present in the payroll and even simple functions could not be relied upon. Requesting that the problems be addressed, the State sent a 37 page “cure” letter to SAP in Oct 2012. The failure to overcome the concerns in the letter resulted in State Controller John Chiang terminating the contract with SAP on 8th Feb 2013 and announcing that MyCalPAYS would be deactivated so that payroll functions could revert to the legacy systems. Since then the State of California is reported to have initiated a review process to determine if parts of the system can be salvaged for a third attempt and whether or not litigation is required.

Contributing factors as reported in the press:
Early reports do not yet indicate the cause of the problems. The project appears to have a meaningful governance structure in place and a supplier with significant prior experience. The fact that the payroll system had so many problems and the fact that such projects do generally have access to prior historical data that can be used to verify test results indicates that quality control / testing issues exist. Additional notes will be added should more information become publicly available.

Reference links :

1. Project termination announcement from the State Controllers Office - 8 Feb 2013
2. California ends contract with SAP over troubled IT project

While understanding the causes of project failure is important, without a common definition of “success”, there is no clear basis for differentiating a success from a failure. Clearly none of the projects in the Catalogue of Catastrophe can be regarded as great successes (some resulted in bankruptcies, many were cancelled before completion and some caused significant damage to public relations). However, those examples represent the extreme of project failure and in practice, there is a sliding scale between total failure and absolute success. Given that our definition of success should be the guiding light towards which projects are focused, I think it important that we have a clear definition of what constitutes “success” .

On the surface you might think that defining success would be relatively simply, but in practice different people define success in different ways. Based on discussions with a wide variety of project participants and observation of people’s actual behaviours in real-life projects, I’ve classified the different definitions into five tiers:

• Tier 1 – The project was a success if it delivers all or most of what it said it would (the scope), regardless of schedule or budget performance
• Tier 2 – The project was a success if it delivers what it said it would, on schedule and/or within the agreed budget
• Tier 3 – The project was a success if it delivers what it said it would, on schedule, within the agreed budget and to the expected quality standards
• Tier 4 – The project was a success it delivers on all agreed project objectives, be they scope, schedule, budget, quality or outcomes based (i.e. goals to be achieved or strategic positions to be attained)
• Tier 5 – The project was a success if the product produced by the project creates significant net value for the organization after the project is completed.

The classic textbook definition says that a project is a success if it achieves all of the agreed “project objectives” (i.e. the tier 4 definition). However, perhaps because schedule and budget are the most visible dimension, in the midst of a project many people behave as if either tier 2 or 3 where the definitive answer. Once a project is over and once the product produced by the project has time to be used, the perspective sometimes changes and people (especially the Sponsor, members of the public and the media) often look to the tier 5 definition when making their final retrospective judgment.

These differing perspectives mean that a project’s relative degree of success of failure may change over time. The Sydney Opera house is a good example. The original 1957 project plan called for the project to be finished in 5 years at a cost of $7M. In the end, the project cost $110M and took 13 years. By any measure that was a severely troubled project and at the time, the press savaged the project for its many missed deadlines and projections. In retrospect, now that we can look back many years later, it is clear that the project created something of enormous value. Today the Opera house is an icon of Australia and a magnet that draws tourists to Australia. It seems reasonable that those tourist dollars will have paid back the development costs many times over and I doubt there are many people who would now say the project should never have been done.

Figure 1 – Layers in project success
click for larger image

One reason people have difficulty agreeing on a definition is because there are two interconnected dimensions in which project success or failure is judged (Figure 1). There is “project management success” (i.e. delivering in accordance with the agreed project objectives) and there is “product success” (i.e. the amount of value the project’s deliverables bring once the project is over). Those two would equate to the tier 4 and 5 definitions respectively. With that separation in place, it is fair to say that the Sydney Opera house was a successful product born from a deeply troubled project.

When you ask Project Sponsors which definition of success is most important to them, they usually pick the tier 5 definition (creating real value). Given that the Sponsors are usually the ones paying for the project (i.e. the customer) it then makes sense to me that the tier 5 definition should be the starting point when planning a project. That doesn’t mean that schedule and budget aren’t important, they are! But the creation of value should be the context within which decisions are made, within which plans are set and against which progress is judged. That approach takes quite a mature environment and many organizations would need to look long and hard at their corporate culture if they were to try and adopt it in any meaningful way. It requires us to make value based decisions when facing trade-offs and at times it may mean prioritizing value over schedule and budget.

Of course getting agreement about how to define success is not easy. While Project Sponsors do usually think about projects in terms of value created, some Project Managers dislike any definition of success that encompasses value created. Perhaps fearing that they will be held accountability for things over which they have no direct control, some in the Project Management community prefer the tier 3 or 4 definitions. These fears are certainly understandable. Many of the key decisions that influence the value created are out of the Project Manager’s direct control. If you are hired to build condos and you deliver on schedule and on budget, but the architect designed units that are not appealing in the market or the developer over prices the units, it’s hard to swallow the bitter pill and call the project a failure. While that is an understandable perspective, my thinking is that the definition of success or failure should be from the customer’s perspective rather than from that of the person hired to manage the delivery of the project. To me, doing it the other way round is a little like saying that the sun revolves around the earth (i.e. that “Project Management” success is more important than “product” success).

As a professional Project Manager I think it is important to have a clear picture of what success means and I personally choose to use the tier 5 definition as my guide. That definition frames the context within which I make decisions and how I guide my clients. Given the importance of the issue it is a subject I encourage all organizations to think about. If the organization’s definition of success is wrong then the context within which decisions are made will be wrong and that can easily become the trigger that leads to a failure. Similarly if stakeholders have different definitions of success we can end up with different people pulling the project in different directions.

Now I’m fully aware that some in the Project Management community will disagree with the use of the tier 5 definition. Some will say that the project ends when the project closes and that moment in time is the point at which success should be judged. While I respect their perspective, it is not me that they will have to convince. The perception of the public, the Sponsors and the media are the forum in which success or failure is judged. While we may like to impose a very specific point in time when success should be judged, it is doubtful anyone would be able to influence how people outside of the Project Management community make their judgment. Trying to change public perception would be an impossible battle and it is not one I would choose to fight.

Editor: Robert Goatham