CIO Perspectives

The rollout: What should we learn?

Richard Spires

The troubled launch of pains me – as someone who has great passion for wanting to make government IT more effective, this public spectacle once again casts federal IT in a very negative light. As a federal IT community we appear unempowered, and worse, incompetent.

My observations here are based solely on public information I have gleaned through the media and listening to the various congressional hearings. I was never close to the planning or development of the website and supporting back-end systems. In full disclosure, however, I did participate in one planning session a couple of years ago when I was Department of Homeland Security CIO. The session was to ensure various agencies (including DHS) identified the individuals to work with the Centers for Medicare and Medicaid Services on the required data-sharing to support the enrollment process.

CIO Perspectives is a new FCW feature by former Department of Homeland Security CIO Richard Spires. The column originally intended as Spires' debut will appear mid-November both online and in print; this reaction to the rollout serves as a web-only preview.

A significant part of my 30-year career has been devoted to IT program management and oversight. For a system as complex as, best practice would have led to a plan that included: 

1) Completion and testing of all subsystems six months prior to public launch.

2) Three months of end-to-end functional integration testing.

3) Concurrent performance testing that would have simulated loads up to 10 times greater than expected (especially since it was difficult to model expected peak loads).

4) A subsequent three-month pilot phase in which selected groups of users were using the system to work off problems not caught in testing.

While I do not know for certain, I would expect that CMS had original development plans that were close to best practice. Yet some of the contractors involved have admitted that there were only two weeks of end-to-end integration testing prior to launch. That means the American public is serving as the integration testers of this system – not a situation anyone would ever plan for or want.

So what happened? There is pattern recognition for those of us who have been involved in many large IT programs. First, it is difficult to accurately plan the level of effort and time to develop new systems that are composed of complex and interdependent subsystems. Hence, there should have been schedule management reserve built in up-front, at least three months and perhaps as much as six months.

Second, given that different contractors were responsible for different subsystems, there needed to be a strong and competent program management office (PMO) that oversaw the program and the integration of the subsystems into a coherent, functional system. The evidence suggests that the PMO was not nearly as effective as required.

Third, the launch date of Oct. 1 was deemed immovable. As development schedules slipped, as integration challenges mounted, there were clearly compromises made so as not to delay the launch. I suspect little functionality could be deferred (the site must enable the full enrollment process), so what was compromised is good practice. It is simply bad practice to launch a complex system with very little end-to-end testing. There is no excuse for this, and given the complexity of systems CMS operates, there are clearly individuals in CMS who knew this launch would not go well because of inadequate testing.

Finally, there is the biggest failure, and the one that dooms many IT projects: The correct roles and authorities were not assigned to the business and the IT organizations. (In this case the business organization would have included leaders from CMS, HHS and possibly the White House).  

When I review an IT program, that role assignment and the authorities are one of the first things I look at to assess the health of that program. The evidence on the launch of shows clearly the balance was not correct. As reported by the media, a change in a requirement that disabled the ability for users to browse insurance policies without first enrolling was made just two weeks before launch. This was much too late -- requirements should have been locked down months before then. The business organization had the ability to make changes that led to bad program management practice.

In subsequent columns I plan to address a number of issues and recommendations regarding large IT program management. But for now, let's focus on how to address the proper partnership between the business and IT.

There must be a program governance model in place that recognizes the proper roles and authorities of the business organization and the IT organization for there to be success.  The business organization must be intimately involved in helping define requirements, making hard functionality trade-offs, and being a champion for the program with stakeholders both inside and outside the agency. The IT organization must establish a solid PMO with appropriate use of best practices to deliver large IT programs. And there needs to be a regular forum in which the business organization and IT organization executives work together to help the PMO make the tough decisions in running a program.

A fundamental tenet, however, is that sound program management practice must always be followed. There are no shortcuts to delivering large, complex IT programs. Having been intimately involved in dozens of such programs, I can state with absolute certainty that executing with anything less than solid program management practice is very high risk and leads to failures. The administration would be well served to incorporate the proper governance model for all large, complex IT programs.

One last point: The team of government personnel and contractors correcting must be working tremendously long hours and are under tremendous pressure. I thank them for their efforts.

About the Author

Richard A. Spires has been in the IT field for more than 30 years, with eight years in federal government service. He served as the lead for the Business Systems Modernization program at the IRS, then served as CIO and deputy commissioner for operations support, before moving to the Department of Homeland Security to serve as CIO of that agency. He is now CEO of Learning Tree.

Cyber. Covered.

Government Cyber Insider tracks the technologies, policies, threats and emerging solutions that shape the cybersecurity landscape.


Reader comments

Sun, Dec 1, 2013 Jed

These lessons are very basic fundamental objectives in a strategic sense but do not fundamentally address the issues around today's Application Development Lifecycle Management and its transition into a monitored Ops strategy. At the core of all this, the problems surround the complexities in developing Web strategies in today's cloud and the approach to addressing those complexities gear up what I call the 'siloed approach', or rather hiring experts in each area of technology and then compartmentalizing their efforts to lead to a finished project. This type of approach used to work for the traditional style of IT, but the paradigm is shifting obviously due to the increasingly evolving demands and expectations by end-users. The bottom line for the lesson here is that they could have found a vendor that had testing tools enable the development of the website to meet the strategic objectives and be able to seamlessly and rapidly deploy in heterogeneous IT environments (and even hybrid cloud). Perhaps this sounds forward thinking, but the technology and tools to do such already exists and many use cases and references are available. Perhaps the article is trying to point to a more apparent lesson would be to learn from is that the nature of politics itself drives IT in a fashion that is inefficient, but a cohesiveness process or function is necessary to achieve the objectives.

Wed, Nov 6, 2013

As someone else pointed out, these "lessons" are really first grade stuff - things you would know in software dev 101. Things this basic should not be a lesson we have to learn from a botched deployment that's causing so many problems for millions of Americans. It's my undertsanding that CMS was the systems integrator for the project. To me, that seems to be a fatal flaw for this project as well. Why was CMS serving as the systems integrator, if it did not have the extremely basic IT skills to know that testing and PMO support are important? The government would have been better off setting up the procurement for this project so that a commercial entity that specializes in IT and software development was in charge as the general contractor from the get go - which is where it has now ended up. The level of mismanagement and incompetence that led to collossal failure of this scale is simply hard to believe - this is one for the textbook.

Tue, Nov 5, 2013 Dick Spires

For everyone in IT and BIG Systems. Please understand and correct one of the above posts which confused the role of DHS and HHS (the Contractor for Integration via CMS). DHS was only a data provider NOT the Contractor and/or the Integrator for the Web Site! That was clearly the responsibility of HHS (via CMS). Please don't confuse these major responsibilities. To do so creates" spreading the blame" to those within DHS which had NO responsibility for Integration plus end-to end testing. In the sense of being correct, transparent, plus fairness - Please correct your posting comments ASAP. Thank you for your timely response.

Tue, Nov 5, 2013

At some point they will find the person or persons who were genuinely concerned about the way the project was going and told their superiors only to be told "there, there everything is going to be just fine." Congress needs to find these people and talk to them. Then the guilty can be identified and encouraged to resign/retire.

Tue, Nov 5, 2013

To words: Byzantine and Balkanized. In general the government’s approach is extremely hierarchical and compartmentalized (aka stovepipe). Its’ my experience as the size and profile of a program or project grows so does the levels and dysfunction in the chain of command. Add to this every significant program becomes a means to accomplishing a personal agenda. Information is filtered for the purpose of self-aggrandizement not program performance. Many are not managing a program they are managing their way upward. In the end not only does program performance suffer it creates a culture that effects ongoing individual performance. Furthermore, it’s often the case the most pandering of these sycophants is promoted up and out long before the proverbial stuff hits the fan. Often to criticize the efforts they were so negatively a part of. On the other side the way the government goes about procurement is a charade, to say the least. It’s a fraternity of bidders who know how to play a game. It operates under the pretense of fairness and competitiveness but its primary purpose is to funnel tax dollars back into congressional districts through corporate hands. In many cases the legislature is more to blame then the executive branch. That’s the business model. It can only be considered perforce based in that it rewards poor performance. DHS’s implementation of is a microcosm, an epitome of an all too common case of how the government does business. This was not a risky R&D effort with a lot of unknowns, it would have benefited greatly from agile development. However, given the whole procurement system is one huge monolithic cathedral the results were sadly a forgone conclusion. How can other central governments in the world pull off similar efforts at a fraction of the cost so effectively? The argument that the size and scope of our systems is so much greater is vacuous. was an initial flop because it failed to meet its non-functional requirements. If DHS had managed these scaling the system would not have been the nightmare that it turned out to be. The problems here are not technical.

Show All Comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group