Health IT

Consultants warned administration about flawed business plan

Henry Chao, testifying before the House Oversight and Government Reform Committee in November 2013

Henry Chao, shown here before the House Oversight and Government Reform Committee in early November, on Nov. 19 gave the House Energy and Commerce Committee an upbeat assessment of security testing for

Senior officials at the White House and the Department of Health and Human Services were warned as early as March that a flawed business plan for building could lead to critical problems with the site’s launch on Oct. 1.

Consulting firm McKinsey and Co. prepared a report for the Centers for Medicare and Medicaid Services (CMS) that was highly critical of the site’s development plan. Among the chief concerns was the extension of the policy planning and requirements phase into the period devoted to designing, building and testing the system.

The report also cited the lack of a clear decision-maker for the project and said the lack of an “end-to-end operational view of interdependencies” could lead to problems with integrating elements built by the more than 50 contractors that worked on

McKinsey interviewed CMS technology staffers and other federal employees who worked on the project and reviewed planning documents and reports. The firm presented its findings in a series of briefings to top officials at CMS, HHS and the White House.

Republicans on the House Energy and Commerce Committee released an encapsulated version of the document’s findings as evidence that top officials were alerted to the potential for serious problems six months before the launch of

A CMS spokesperson said the review was “part of a standard process to identify potential risks and develop mitigating strategies.”

According to McKinsey, a project of’s magnitude should be fully scoped before the design phase begins. Sequential testing and revision should occur between the design and build phases, and after the site is built it should be tested before gradually being launched into a live environment. McKinsey’s analysis indicates that the plan to fully launch the site on a single date constituted a risk. Its recommendations included finalizing open requirements by the end of April, agreeing on performance metrics and putting new governance in place.

Although not all the potential problems McKinsey identified came to pass, the analysis gives weight to the assessment that the process was overly compressed and poorly designed. At a Nov. 18 Software and Information Industry Association event, former e-government leader Mark Forman said the plan for “made a mockery of modular procurement” because the pieces of the system didn’t fit together properly. The vendor management process was also broken, he added.

“Flags were definitely raised throughout the development of the website, as would be the case for any IT project this complex,” White House spokesman Eric Schultz told the Washington Post. “But nobody anticipated the size and scope of the problems we experienced once the site launched.”

McKinsey interviewed CMS Deputy CIO Henry Chao for the report, but in testimony Nov. 19 before the Energy and Commerce Oversight and Investigations Subcommittee, Chao said he was not among the officials briefed on the firm’s findings. He added that he was not aware of any changes in operations or strategy that were made in response to McKinsey’s report.

“My direction from [CMS Administrator] Marilyn Tavenner was to deliver a system on Oct. 1,” Chao said, and a system was indeed delivered Oct. 1, although he was not prepared to defend its performance at launch.

Security Concerns

The hearing was convened to discuss’s possible security risks. The committee heard testimony from several security experts who are not affiliated with about possible points of vulnerability. The firm TrustedSec pointed out several ways bad actors could use spoofing to redirect users to sites designed to look like in order to compromise their personal information.

Chao said that despite concerns about the lack of end-to-end security testing of the site, the pieces of the system that went live Oct. 1 met federal security standards, and the patches being made under the ongoing “tech surge” were subject to testing as well.

The overall system was subject to monitoring for intrusions, breaches and security flaws. “As of today,” Chao said, “no vulnerabilities identified by our testing have been exploited by an attack.”

His upbeat assessment of the system’s security was backed up by a second panel of witnesses that included security contractors and testers who said the site was being protected and monitored at standards beyond what is required under federal law.

The seeming gap between incomplete security control assessments and the temporary authority to operate granted to the marketplace where consumers shop for insurance reveals how much of the system has yet to be built or go online. Chao said 30 percent to 40 percent of the site has yet to be completed. For instance, the financial management system is due to be released in December, but it is not finished and has not been subjected to security testing.

CMS spokesperson Julie Bataille said that piece of the site processes payments to carriers and does not affect how individuals make payments. Those backend tools are not essential until mid-January of next year, she added.

The marketplace is a key part of the overall system. It reconciles enrollment reports with insurance carriers, handles premium processing, assesses and collects carriers’ fees, and pays the premium subsidies for qualified insurance customers. McKinsey’s report identified the financial management system as a potential trouble spot because of limited testing and resources and the heavy emphasis being placed on enrollment.

Chao, a team of contractors led by Quality Software Services Inc. and the tech surge effort spearheaded by former Obama administration official Jeff Zients are working to fix so that it operates smoothly for about 80 percent of users by the end of November.

In a call with reporters, Zients said the tech surge has made measurable progress in fixing the site. Error rates for users are down to about 1 percent, and the site is able to support an increasing volume of users. The team has completed about 200 items from its list of fixes and is turning its attention this week to 50 additional updates and improvements.

“I think that’s an attainable goal given what I’ve seen so far,” Chao said of the Nov. 30 deadline, but added that he could give no guarantees that the goal would be met.

About the Author

Adam Mazmanian is executive editor of FCW.

Before joining the editing team, Mazmanian was an FCW staff writer covering Congress, government-wide technology policy and the Department of Veterans Affairs. Prior to joining FCW, Mazmanian was technology correspondent for National Journal and served in a variety of editorial roles at B2B news service SmartBrief. Mazmanian has contributed reviews and articles to the Washington Post, the Washington City Paper, Newsday, New York Press, Architect Magazine and other publications.

Click here for previous articles by Mazmanian. Connect with him on Twitter at @thisismaz.


  • Government Innovation Awards
    Government Innovation Awards -

    Congratulations to the 2021 Rising Stars

    These early-career leaders already are having an outsized impact on government IT.

  • Acquisition
    Shutterstock ID 169474442 By Maxx-Studio

    The growing importance of GWACs

    One of the government's most popular methods for buying emerging technologies and critical IT services faces significant challenges in an ever-changing marketplace

Stay Connected