Cloud failure temporarily crashes

concept art for cloud services is back online after a network failure at Verizon's Terremark cloud service took the site and its main data hub down Oct. 27.

"Verizon Terremark successfully resolved the issue with the networking component overnight, and as of 7 a.m. ET this morning the Data Services Hub was fully operational," HHS spokeswoman Joanne Peters said in a statement. "The technical team continued troubleshooting one issue with the online account creation process in the application and has now opened the online application and enrollment tools back up to consumers."

The outage was the latest in a series of technical glitches for the site, including long waits for service for users and problems in delivering information to insurance carriers on the back end.

However, this was the first problem caused by a cloud service provider.

The failure originated in a Verizon Terremark data center and likely affected other companies that purchase computing power from the tech giant, though Verizon has thus far not responded to media inquiries on the cause of the problem.

It is also unclear how many potential customers attempted to reach the site while it was down only to receive a message that said: "We are experiencing technical difficulties and hope to have them resolved soon. Please try again later."

The outage affected both the Federally Facilitated Exchange which serves insurance customers in 36 states, and the individual exchanges in 14 states and the District of Columbia. The state-based health insurance exchanges as well as the federal system, are dependent on the data hub to operate properly. is the front door to a system for registering for insurance and shopping for policies under the 2010 health care overhaul. Since launching on Oct. 1, it has been plagued by errors and long waits for service on the user interface side, and problems in delivering information to insurance carriers on the back end. This is the first reported outage related to a failure at a cloud provider.

Separately, the website was redesigned over the weekend to remove a picture of a smiling woman, which was replaced with navigation icons that show users how to apply by phone, in person and by mail, in addition to online.

The administration is encouraging applicants to use these alternative enrollment methods while contractors focus on repairing software bugs and capacity issues. This effort is being led by Jeff Zients, a health care entrepreneur and former acting director of the Office of Management and Budget. The administration is promising that online enrollment will work as advertised for the vast majority of users by the end of November. 

CMS did announce that it had knocked a few items off the "punch list" being managed by Zients. The load times for comparing insurance plans have dropped considerably, from minutes to seconds. Also, eligibility notices which had been garbled now appear as designed to shoppers at the end of the application process. is still delivering faulty enrollment information to insurance carriers. A fix to the delivery of enrollment forms, called 834s in the industry, remains a high priority for the team working on, but there's no work on when it will be completed. The issue "is on the top of the punch list," CMS spokeswoman Julie Bataille said, adding that CMS is having ongoing conversations with insurance issuers, "to identify , diagnose, and prioritize issues as we work through our punch list of fixes."

Bataille also acknowledged that CMS Administrator Marilyn Tavenner was in charge of implementing the IT pieces and other aspects of the 2010 health care overhaul. Tavenner is scheduled to testify before a hearing of the House Ways and Means Committee on Oct. 29 about the buggy launch of However, CMS is still declining to name the technology specialists inside government and from the private sector who were tapped to lead a "tech surge" to identify problems with the software and the performance of The group is thought to include several members of the Presidential Innovation Fellows team, as well as Federal CTO Todd Park.

It's not known which private sector firms outside of the contractors working on the project are pitching in, or whether they are getting paid. Rep. Darrell Issa (R-Calif), chairman of the House Oversight and Government Reform Committee, is threatening to issue subpoenas to find out the identities of the team behind the tech surge, and has also sent letters to leading technology firms inquiring whether they have detailed any staff to work on

The highly public failure of is attracting a lot of attention in the federal IT community.

The American Council for Technology and the Industry Advisory Council (ACT-IAC) is holding its Executive Leadership Conference in Williamsburg, Va. While government CIOs and other IT executives aren't commenting for the record on, it is clear that there is an appetite to change some aspects of federal IT contracting and development.

"We do not have an 18-month or a two-year cycle to produce software based on requirements that we gather and develop in silos and that we pull together toward the end," said Lena Trudeau, associate commissioner of GSA's Office of Strategic Innovations in an Oct. 27 speech. While she was not directly addressing, she spoke in favor of close collaboration between agencies and IT contractors, and regular peer review of software code.

About the Authors

Adam Mazmanian is executive editor of FCW.

Before joining the editing team, Mazmanian was an FCW staff writer covering Congress, government-wide technology policy, health IT and the Department of Veterans Affairs. Prior to joining FCW, Mr. Mazmanian was technology correspondent for National Journal and served in a variety of editorial roles at B2B news service SmartBrief. Mazmanian started his career as an arts reporter and critic, and has contributed reviews and articles to the Washington Post, the Washington City Paper, Newsday, Architect magazine, and other publications. He was an editorial assistant and staff writer at the now-defunct New York Press and arts editor at the online network in the 1990s, and was a weekly contributor of music and film reviews to the Washington Times from 2007 to 2014.

Click here for previous articles by Mazmanian. Connect with him on Twitter at @thisismaz.

Frank Konkel is a former staff writer for FCW.

The Fed 100

Read the profiles of all this year's winners.


  • Then-presidential candidate Donald Trump at a 2016 campaign event. Image: Shutterstock

    'Buy American' order puts procurement in the spotlight

    Some IT contractors are worried that the "buy American" executive order from President Trump could squeeze key innovators out of the market.

  • OMB chief Mick Mulvaney, shown here in as a member of Congress in 2013. (Photo credit Gage Skidmore/Flickr)

    White House taps old policies for new government makeover

    New guidance from OMB advises agencies to use shared services, GWACs and federal schedules for acquisition, and to leverage IT wherever possible in restructuring plans.

  • Shutterstock image (by Everett Historical): aerial of the Pentagon.

    What DOD's next CIO will have to deal with

    It could be months before the Defense Department has a new CIO, and he or she will face a host of organizational and operational challenges from Day One

  • USAF Gen. John Hyten

    General: Cyber Command needs new platform before NSA split

    U.S. Cyber Command should be elevated to a full combatant command as soon as possible, the head of Strategic Command told Congress, but it cannot be separated from the NSA until it has its own cyber platform.

  • Image from Shutterstock.

    DLA goes virtual

    The Defense Logistics Agency is in the midst of an ambitious campaign to eliminate its IT infrastructure and transition to using exclusively shared, hosted and virtual services.

  • Fed 100 logo

    The 2017 Federal 100

    The women and men who make up this year's Fed 100 are proof positive of what one person can make possibile in federal IT. Read on to learn more about each and every winner's accomplishments.

Reader comments

Tue, Oct 29, 2013

Read the article, commenters. This was a outage in the networking services of cloud Infrastructure as a Service (Iaas) provided to the government by a commercial vendor. It affected multiple customers not just "Government requirements" for "programming" have nothing to do with this when it's a networking outage. "Good IT professionals (contractors)" would know the difference. And most major commercial cloud providers have had service outages or failover problems after launch. This points to issues with the commercial cloud provider's service, not the government's system.

Tue, Oct 29, 2013

So a network problem at a Good IT professionals (contractors" facility goes down and it is a gov't requirement problem?lol

Tue, Oct 29, 2013

please do you know the location of the affected Terremark Data Center?

Tue, Oct 29, 2013

Likely the problem is poor govt requirements. Good IT professionals (contractors) will program to requirements. Requirements definition is VERY diffifult and not something the govt does well, but they are in the driver's seat. The old saying, "garbage in, garbage out" comes to mind here. The govt shovels in garbage requirements and guess what comes out......?

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group