The following entry is a record in the “Catalogue of Catastrophe” – a list of failed and troubled projects from around the world.
2012 Mitt Romney Presidential Campaign – USA
Project name : Orca
Project type : Political campaign – Operations management system
Date : Nov 2012
Cost : Unknown
Synopsis :
The 2012 US Presidential election illustrates how the failure to properly test a key system can cause an embarrassing failure at a critical point in time. To help manage the logistics of their election day “get out the vote” push, both the Obama and Romney campaigns developed “operational management” systems that would provide real time tracking of voter turnout in the key battleground states. Providing critical insight into what was happening in the field, the systems fed real time data from volunteers at the polling stations back to campaign headquarters. Allowing the campaign to optimize the use of the available field workers, the systems provided the data needed to ensure volunteers were directed to those precincts and counties where they were needed most.
With smart-phone enabled volunteers at each polling station, the systems channeled streams of real-time data back to campaign headquarters. Crunching the numbers the systems gave their respective campaigns the “big picture” view as the day progressed. With the systems up and running the campaign had information and control. If the system failed there they were in the dark having to piece things together based on hundreds of phone conversations.
According to available reports the Obama system (called “Narwhal”) was thoroughly tested before the big day. Dress rehearsals were conducted weeks in advance and the team developed procedures for every possibly mode of failure they could think of. On the day, those investments paid off and the system functioned as planned.
At Romney headquarters the story was different. Users of the system (called “Orca”) reported outages, slow responses and other technical issues that prevented them using the system effectively. At times connectivity to the field workers was lost and according to reports, at one point the Internet Service Provider (ISP) connecting the system to the internet shut down access as they thought the high level of traffic was caused by a “denial of service” type attack. Frustration from the users rose rapidly and trying to fix the issues took critical resources away from managing the campaign.
The day after the election, stories about the system began to surface. At the core of the problem was a failure to subject the system (or it’s users and support personnel) to the same level of testing as Narwhal had received. There had been no full dress rehearsals and election day was the first time Orca had been run on the full set of systems infrastructure used at Romney’s Boston headquarters. In addition, reports indicated that the campaign had failed to communicate with their ISP and as a result the ISP was not expecting the high volume of traffic that would be flowing.
Would Orca have changed the outcome of the election had it worked more smoothly? Probably not. Obama’s margin of victory was significant and aggregated polling sites (such as Votamatic and Nate Silver’s New York Times FiveThirtyEight blog) had been predicting the precise margin of victory for some time in advance of the polls. The Orca story is however a reminder to all organizations of the need to ensure they do thoroughly test critical operational systems before releasing them into their live environments. Failure to do so can dramatically raise stress levels at critical points in time, divert resources away from core business functions and lead to embarrassing public relations failures.
Contributing factors as reported in the press:
Failure to perform dress rehearsals. Failure to plan for peek performance system requests. Failure to stress test a new system in its full operational environment. Failure to inform key stakeholders. Lack of risk management.
Reference links :
- When-the-nerds-go-marching-in
- Project Orca – Wikipedia page