Health IT

Consultants warned administration about flawed business plan

Henry Chao, testifying before the House Oversight and Government Reform Committee in November 2013

Henry Chao, shown here before the House Oversight and Government Reform Committee in early November, on Nov. 19 gave the House Energy and Commerce Committee an upbeat assessment of security testing for

Senior officials at the White House and the Department of Health and Human Services were warned as early as March that a flawed business plan for building could lead to critical problems with the site’s launch on Oct. 1.

Consulting firm McKinsey and Co. prepared a report for the Centers for Medicare and Medicaid Services (CMS) that was highly critical of the site’s development plan. Among the chief concerns was the extension of the policy planning and requirements phase into the period devoted to designing, building and testing the system.

The report also cited the lack of a clear decision-maker for the project and said the lack of an “end-to-end operational view of interdependencies” could lead to problems with integrating elements built by the more than 50 contractors that worked on

McKinsey interviewed CMS technology staffers and other federal employees who worked on the project and reviewed planning documents and reports. The firm presented its findings in a series of briefings to top officials at CMS, HHS and the White House.

Republicans on the House Energy and Commerce Committee released an encapsulated version of the document’s findings as evidence that top officials were alerted to the potential for serious problems six months before the launch of

A CMS spokesperson said the review was “part of a standard process to identify potential risks and develop mitigating strategies.”

According to McKinsey, a project of’s magnitude should be fully scoped before the design phase begins. Sequential testing and revision should occur between the design and build phases, and after the site is built it should be tested before gradually being launched into a live environment. McKinsey’s analysis indicates that the plan to fully launch the site on a single date constituted a risk. Its recommendations included finalizing open requirements by the end of April, agreeing on performance metrics and putting new governance in place.

Although not all the potential problems McKinsey identified came to pass, the analysis gives weight to the assessment that the process was overly compressed and poorly designed. At a Nov. 18 Software and Information Industry Association event, former e-government leader Mark Forman said the plan for “made a mockery of modular procurement” because the pieces of the system didn’t fit together properly. The vendor management process was also broken, he added.

“Flags were definitely raised throughout the development of the website, as would be the case for any IT project this complex,” White House spokesman Eric Schultz told the Washington Post. “But nobody anticipated the size and scope of the problems we experienced once the site launched.”

McKinsey interviewed CMS Deputy CIO Henry Chao for the report, but in testimony Nov. 19 before the Energy and Commerce Oversight and Investigations Subcommittee, Chao said he was not among the officials briefed on the firm’s findings. He added that he was not aware of any changes in operations or strategy that were made in response to McKinsey’s report.

“My direction from [CMS Administrator] Marilyn Tavenner was to deliver a system on Oct. 1,” Chao said, and a system was indeed delivered Oct. 1, although he was not prepared to defend its performance at launch.

Security Concerns

The hearing was convened to discuss’s possible security risks. The committee heard testimony from several security experts who are not affiliated with about possible points of vulnerability. The firm TrustedSec pointed out several ways bad actors could use spoofing to redirect users to sites designed to look like in order to compromise their personal information.

Chao said that despite concerns about the lack of end-to-end security testing of the site, the pieces of the system that went live Oct. 1 met federal security standards, and the patches being made under the ongoing “tech surge” were subject to testing as well.

The overall system was subject to monitoring for intrusions, breaches and security flaws. “As of today,” Chao said, “no vulnerabilities identified by our testing have been exploited by an attack.”

His upbeat assessment of the system’s security was backed up by a second panel of witnesses that included security contractors and testers who said the site was being protected and monitored at standards beyond what is required under federal law.

The seeming gap between incomplete security control assessments and the temporary authority to operate granted to the marketplace where consumers shop for insurance reveals how much of the system has yet to be built or go online. Chao said 30 percent to 40 percent of the site has yet to be completed. For instance, the financial management system is due to be released in December, but it is not finished and has not been subjected to security testing.

CMS spokesperson Julie Bataille said that piece of the site processes payments to carriers and does not affect how individuals make payments. Those backend tools are not essential until mid-January of next year, she added.

The marketplace is a key part of the overall system. It reconciles enrollment reports with insurance carriers, handles premium processing, assesses and collects carriers’ fees, and pays the premium subsidies for qualified insurance customers. McKinsey’s report identified the financial management system as a potential trouble spot because of limited testing and resources and the heavy emphasis being placed on enrollment.

Chao, a team of contractors led by Quality Software Services Inc. and the tech surge effort spearheaded by former Obama administration official Jeff Zients are working to fix so that it operates smoothly for about 80 percent of users by the end of November.

In a call with reporters, Zients said the tech surge has made measurable progress in fixing the site. Error rates for users are down to about 1 percent, and the site is able to support an increasing volume of users. The team has completed about 200 items from its list of fixes and is turning its attention this week to 50 additional updates and improvements.

“I think that’s an attainable goal given what I’ve seen so far,” Chao said of the Nov. 30 deadline, but added that he could give no guarantees that the goal would be met.

About the Author

Adam Mazmanian is executive editor of FCW.

Before joining the editing team, Mazmanian was an FCW staff writer covering Congress, government-wide technology policy, health IT and the Department of Veterans Affairs. Prior to joining FCW, Mr. Mazmanian was technology correspondent for National Journal and served in a variety of editorial at B2B news service SmartBrief. Mazmanian started his career as an arts reporter and critic, and has contributed reviews and articles to the Washington Post, the Washington City Paper, Newsday, Architect magazine, and other publications. He was an editorial assistant and staff writer at the now-defunct New York Press and arts editor at the online network in the 1990s, and was a weekly contributor of music and film reviews to the Washington Times from 2007 to 2014.

Click here for previous articles by Mazmanian. Connect with him on Twitter at @thisismaz.

FCW in Print

In the latest issue: Looking back on three decades of big stories in federal IT.


  • Anne Rung -- Commerce Department Photo

    Exit interview with Anne Rung

    The government's departing top acquisition official said she leaves behind a solid foundation on which to build more effective and efficient federal IT.

  • Charles Phalen

    Administration appoints first head of NBIB

    The National Background Investigations Bureau announced the appointment of its first director as the agency prepares to take over processing government background checks.

  • Sen. James Lankford (R-Okla.)

    Senator: Rigid hiring process pushes millennials from federal work

    Sen. James Lankford (R-Okla.) said agencies are missing out on younger workers because of the government's rigidity, particularly its protracted hiring process.

  • FCW @ 30 GPS

    FCW @ 30

    Since 1987, FCW has covered it all -- the major contracts, the disruptive technologies, the picayune scandals and the many, many people who make federal IT function. Here's a look back at six of the most significant stories.

  • Shutterstock image.

    A 'minibus' appropriations package could be in the cards

    A short-term funding bill is expected by Sept. 30 to keep the federal government operating through early December, but after that the options get more complicated.

  • Defense Secretary Ash Carter speaks at the TechCrunch Disrupt conference in San Francisco

    DOD launches new tech hub in Austin

    The DOD is opening a new Defense Innovation Unit Experimental office in Austin, Texas, while Congress debates legislation that could defund DIUx.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group