VA revisits data consolidation plan

Physicians detail clinical system meltdown

When physicians at 17 Veterans Affairs Department hospitals were unable to log on to the Veterans Health Information Systems and Technology Architecture (VistA) and its Computerized
Patient Record System, they discovered that was not their only problem.

  • VA’s regional data processing center in Sacramento, Calif., did not back up the electronic health records system to the Denver regional processing center as it was supposed to.

  • At four medical centers, backup systems were unavailable or overwhelmed. A read-only backup of patient data was unavailable because of a previously planned, periodic updating of a hospital test account.

  • Physicians relied on health summary data stored on several local PCs.

  • Medical employees had to write discharge instructions, medical notes and medication administration records on paper.

— Mary Mosquera

The Veterans Affairs Department must answer new questions about its data center consolidation strategy after its electronic health records system was unusable for nine hours at 17 VA medical facilities Aug. 31, and backup procedures failed.

VA officials have halted further consolidation moves until they finish evaluating the disruption.

The disruption occurred as VA was moving its health records system, the Veterans Health Information Systems and Technology Architecture (VistA), from local hospital data centers to regional data centers. By creating regional centers, VA officials said they hope to provide better security and standardized management practices to safeguard critical health records and other electronic health information.
VA has two regional data centers serving the West and Northeast regions. Officials planned to open two more regional centers by Dec. 31.

“The root cause of the outage was indeed…human error, more specifically, a failure to adhere to change management procedures.” Jeff Shyshka, Veterans Affairs Department

The VistA disruption affected physicians and patients at VA hospitals, which rely on VistA’s electronic health records for patients’ clinical care. Without VistA, physicians had to revert to paper recordkeeping, which created opportunities for mistakes and slowed the delivery of health services.

“Work to recover the integrity of the medical record will continue for many months since so much information was recorded on paper that day,” said Bryan Volpp, associate chief of staff for clinical informatics at VA’s Northern California Healthcare System.

VA is investigating the incident internally and will hire a company to conduct an independent review to ensure that the department’s contingency plans work, Robert Howard, VA’s chief information
officer, told lawmakers. Howard also is
re-evaluating VA’s regional processing

“We want to examine the whole program and build in more robust backup at the facility level,” Howard said. “We cannot allow hospitals to go down.”

VistA stopped working in northern California because information technology employees did not follow policies and procedures, VA officials said in their initial assessment of the disruption.
“The root cause of the outage was indeed…human error, more specifically, a failure to adhere to change management procedures,” said Jeff Shyshka, deputy assistant secretary of enterprise operations and infrastructure at VA’s CIO Office.

The disruption happened during daytime hours, the busiest time for treating patients at the hospitals. Ben Davoren, director of clinical informatics at VA’s San Francisco Medical Center, related details of the incident during a recent hearing before the House Veterans’ Affairs Committee. He called it “the most significant technological threat to patient safety VA has ever had.”

Shyshka said IT specialists from several VA health care facilities in the region were meeting at the data center. An employee did not follow procedures to vet and implement a system change, he said.
VA resolved the regional data center problem by backing out of a network port configuration change that an employee made earlier that day. That port configuration is the likely cause of the problem, Shyshka said.

“As with any collocation undertaking of this magnitude, there will always be the potential for human error,” Shyshka said. “Ensuring effective communications processes between the teams managing the collocated VistA systems and the IT staff at the local facilities is perhaps the greatest challenge.”

Isolated incident

Charles De Sanno, associate deputy assistant secretary of infrastructure engineering at VA’s CIO office, said he thinks what happened is an isolated incident. “I believe a mistake was made.” De Sanno is one of the architects of the regional data processing center program. He is also executive director of VA’s enterprise infrastructure engineering and northeast operations in New York.
“What you have here is [like] someone crossing the street and not looking and [getting] hit by a car,” De Sanno said. Even if the change the employee made should have been made, it was not authorized, and it was implemented improperly, he said. “We have to ensure that we follow policy and procedure.”

Under VA’s regional processing program, IT systems are physically secured in commercial data centers.

Vendors own and maintain the data center physical plant and network infrastructure and provide support to VA and other customers using the data center.

VA IT professionals operate, manage and maintain the department’s health care information systems remotely, Shyshka said.

About the Author

Mary Mosquera is a reporter for Federal Computer Week.

Rising Stars

Meet 21 early-career leaders who are doing great things in federal IT.


  • SEC Chairman Jay Clayton

    SEC owns up to 2016 breach

    A key database of financial information was breached in 2016, possibly in support of insider trading, said the Securities and Exchange Commission.

  • Image from

    DOD looks to get aggressive about cloud adoption

    Defense leaders and Congress are looking to encourage more aggressive cloud policies and prod reluctant agencies to embrace experimentation and risk-taking.

  • Shutterstock / Pictofigo

    The next big thing in IT procurement

    Steve Kelman talks to the agencies that have embraced tech demos in their acquisition efforts -- and urges others in government to give it a try.

  • broken lock

    DHS bans Kaspersky from federal systems

    The Department of Homeland Security banned the Russian cybersecurity company Kaspersky Lab’s products from federal agencies in a new binding operational directive.

  • man planning layoffs

    USDA looks to cut CIOs as part of reorg

    The Department of Agriculture is looking to cut down on the number of agency CIOs in the name of efficiency and better communication across mission areas.

  • What's next for agency cyber efforts?

    Ninety days after the Trump administration's executive order, FCW sat down with agency cyber leaders to discuss what’s changing.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group