The perils of petascale IT projects

Good news: We no longer have to talk about megascale IT projects. Large-scale ventures that typically cost $1 billion or more, megaprojects used to be all the rage, but they are quickly being superseded by petascale IT initiatives. Those projects can cost even more, involve complexity on a truly massive scale and require petaflops of computer processing.

Despite the horrendous track record of delivering on even moderately complex IT projects, public-sector CIOs continue to embrace the design, planning and execution of petascale IT projects. Consider the Department of Homeland Security's SBInet, a component of the Secure Border Initiative. The $8 billion project commenced in 2006 and aimed to place surveillance systems along the southwest border of the U.S. to identify and prevent illegal crossings.

In 2010, SBInet was halted and many of the project's funds were diverted to other border initiatives. The project was riddled with problems, including cost overruns, poorly defined schedules, missed deadlines, inadequate oversight, an unclear strategy on how to actually secure the border or manage threats, poor engagement with contractor Boeing, and poor performance on the part of the company.

Calling a petascale IT project complex would be a huge understatement. Such projects not only have numerous stakeholders, they also span multiple time horizons (e.g., election cycles, brief CIO tenures at agencies, etc.). They rely on multiple physical and logical infrastructures -- each of which might be owned, controlled and operated by multiple players -- and they have overly restrictive and detailed budget allocations that prevent quick changes in response to events.

Furthermore, they are technologically complex from both innovation and design perspectives, and there is limited ability for leaders to learn from previous efforts because all projects of this scale are unique.

And those are just some of the issues that occur in petascale projects.

The high cost of complexity

Those challenges manifest themselves in the time, effort and cost involved in the projects. Cost overruns in particular are a significant way in which many projects are determined to be a success or failure. One notable megascale IT project that experienced a 560 percent cost overrun was the Medicare Transaction System. In the 1990s, this system was set to modernize Medicare to better serve the 38 million Americans who are over the age of 65 or have a disability.

The project was plagued with unnecessary risks, delays, unmet deliverables, poorly detailed plans, an over-reliance on contractors, fragmentation, and a poor understanding of the existing system and contractor duties.

More recent projects, such as the October 2013 launch of HealthCare.gov, show that IT failures are not an issue of the past. Other failed projects include DHS' software upgrade to its terrorist-tracking system. The $500 million Railhead project had a team of 862 contractors and yet was riddled with architectural and quality flaws.

In addition, the Air Force spent seven years and more than $1 billion to merge 240 legacy systems into 12. Unfortunately, the Expeditionary Combat Support System (ECSS) was halted due to extreme technical glitches and delays. The FBI's Virtual Case File system lost $170 million before being canceled, U.S. Citizenship and Immigration Services went $190 million over budget on an automation effort, and the Internal Revenue Service's Business Systems Modernization took more than 10 years to launch and cost $8 billion.

When the Air Force took on ECSS, the project was expected to automate and streamline logistics operations. It began in 2005 with a deployment date of 2012, and Oracle was contracted to develop the system for $88.5 million. Due to delays, the schedule was pushed back by four years to 2016, and it went from a three-phase to a four-phase project. In 2012, the Defense Department officially ended ECSS, saying it would cost another $1 billion to get the project up to military standards.

In a 2010 study undertaken while ECSS was still being developed, the Government Accountability Office found instances of "unusual logic" that caused activities not to be finished on time and led to the tracking of high-level milestones without a clear program structure. By the time the project was canceled, Air Force officials noted that they had received usable hardware and software worth less than $150 million.

According to GAO, 27 agencies planned to spend about $76 billion on 8,142 IT investments in fiscal 2014 -- $17.2 billion on development and acquisition and $58.9 billion on operations and maintenance. However, that total does not include the IT investments made by 58 independent executive branch agencies (including the CIA) or by the legislative or judicial branches.

Despite the big investments, large-scale projects are failing at an alarming cost to the government and taxpayers. The Office of Management and Budget reported in 2010 that the government had spent more than $600 billion on IT since 2000. However, OMB also reported that the federal government had achieved little productivity from its investments. In fact, IT projects account for a considerably higher number of failures than other types of government megaprojects.

OMB requires federal agencies to regularly provide information on their IT investments, capital asset plans and business cases. Over the years, reports from various agencies have revealed several obstacles to IT success that have wasted and continue to waste taxpayer funds and public resources.

One thing is certain: 99.9999 percent of petascale IT projects will fail. Even if they do end up nominally delivering on their intended objectives, they will not meet user expectations or be completed within budget or time constraints.

Moreover, the fallout from the failures of these projects can be severe -- not only for the IT personnel involved in the effort but for IT projects in other parts of the public sector. IT professionals will be looked upon unfavorably and seen as a liability, consultants can and might lose their jobs, and the overall standing of the IT community in the public sector will be tarnished.

It is no secret that most stakeholders involved in failed projects take a hit, to varying degrees. With HealthCare.gov, government administrators were investigated by congressional oversight committees and some even resigned, most notably Department of Health and Human Services Secretary Kathleen Sebelius. The main contractor involved, CGI Federal, experienced media scrutiny regarding its role in the debacle, and its past public-sector project failures were magnified.

Given this reality, we must ask: Why are we so obsessed -- some might say to the point of being delusional -- in pursuing projects of this scale and scope and hoping they will succeed? The oft-misattributed quote about the definition of insanity being "doing the same thing over and over again and expecting different results" applies here.

Four steps to failure on a grand scale

Over the next few months, we will be studying petascale IT projects in the public sector with the goal of providing some insights into why these beasts are so difficult to tame and what can be done about it. In this initial piece, we explore how political shenanigans contribute to the failure of these projects.

Political activity plays a crucial role in a project's success. First, consider the way in which agencies acquire the money to initiate massive projects. Those funds come from the political bartering that happens among members of the executive and legislative branches. Budgets are allocated largely based on the political capital of agencies and their champions.

When cost overruns occur, Congress must agree to pay additional funds to the contractors to complete the work. When the FBI initiated the Virtual Case File phase of its Trilogy IT modernization project in 2000, it was scheduled to take three years and cost $380 million. At the end of 2002, the project was behind schedule and the FBI asked Congress to increase funding by $123 million. Congress obliged partly because of the 2001 terrorist attacks and the resulting political pressures to pay more attention to anti-terrorism efforts.

Second, politics is a heavy influencer of the way projects are carried out. During the development of the HealthCare.gov website and prior to the 2012 presidential election, the White House reportedly asked the agencies involved in the site's development to avoid committing any website or health insurance exchange ideas to paper so that those ideas could not be stolen and used against the Obama administration before the election. That informal directive would prove to be effective in preventing bad news from being leaked, but it also became detrimental to the development of HealthCare.gov because problems were not acknowledged early enough.

Third, political cycles create a certain atmosphere of change within the public sector. The speed with which political leaders and their focus change can be damaging to projects that last years longer than their champions' tenure at the agency. That makes it difficult to maintain governance of a project throughout its full life cycle. For instance, in 2013, California suspended work on its $200 million MyCalPays system, which was designed to modernize the state's payroll. The project's failure was largely attributed to chronic leadership turnover and lapses in due diligence.

Finally, public-sector managers are often forced to find ways to cope with political dysfunction. Constant partisan clashes and disruptive behavior by politicians force agencies into precarious positions because most of the time, it's more about politics than policy. Recent oversight hearings about HHS' role in the HealthCare.gov rollout and the IRS' review of tax-exempt groups are arguably designed to bring attention to problems for political gain rather than to resolve them. After all, the HealthCare.gov fumble was not the first of its kind, and the project was salvaged soon after its launch. Other projects in recent history have not been so lucky, yet they drew a fraction of the oversight.

Political factors also play a large role in the governance of petascale IT projects. Governance issues are so varied and wide-ranging that finding an appropriate mix is a significant challenge. With proper governance, staff and stakeholders can discharge their duties in a way that allows the organization to function efficiently and effectively. Without it, the organization and the project suffer, and the resulting loss of funds and technological failures are physical manifestations of the lack of governance.

What can be done?

Strong governance of complex, petascale projects is critical for the success of IT projects. We offer five recommendations for CIOs to consider:

1. Overestimate the cost and time required to complete the project by a factor of at least three.

2. Document, document and document the risks inherent in the project and ensure that they are communicated clearly and frequently. Also make sure that the press knows that you called attention to those risks even if the political apparatus chose to ignore them.

3. If at all possible, convince political players to scale down the effort and consider an option-based approach that involves funding each stage of the project with a clause to continue funding if results are achieved and the project is still valuable.

4. Limit the number of petascale IT projects that your agency is involved in or has to support at any given time.

5. Study the lessons learned and postmortem reports from numerous other petascale IT projects as part of your due diligence.

The truth about IT megaprojects is that although some are successful, many are failures. Those failures have been repeated over and over, which leads us to the rather obvious conclusion that a new approach must be taken. Although most public managers cannot do anything about the dysfunctional political climate, they can make efforts to not fall victim to the dysfunction. The rapidly changing technological and political landscape reminds us that there is constant pressure to stay alert and ready to conquer the IT challenges that lie ahead.

We would like to hear from you about governance challenges associated with petascale IT projects that you have encountered. What did they look like up close?

As we learn more through our research, we will share our findings here in FCW. If you are interested in participating in our research project, get in touch via Twitter (@KevDesouza) or KevinDesouza.net.

Note: This article was updated on Sept. 1 to remove reference to Centers for Medicare and Medicaid Services CIO Tony Trenkle resigning. Trenkle retired shortly after the launch of HealthCare.gov for a position in the private sector. 

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.