VA revisits data consolidation plan

Physicians detail clinical system meltdown

When physicians at 17 Veterans Affairs Department hospitals were unable to log on to the Veterans Health Information Systems and Technology Architecture (VistA) and its Computerized
Patient Record System, they discovered that was not their only problem.

  • VA’s regional data processing center in Sacramento, Calif., did not back up the electronic health records system to the Denver regional processing center as it was supposed to.

  • At four medical centers, backup systems were unavailable or overwhelmed. A read-only backup of patient data was unavailable because of a previously planned, periodic updating of a hospital test account.

  • Physicians relied on health summary data stored on several local PCs.

  • Medical employees had to write discharge instructions, medical notes and medication administration records on paper.

— Mary Mosquera

The Veterans Affairs Department must answer new questions about its data center consolidation strategy after its electronic health records system was unusable for nine hours at 17 VA medical facilities Aug. 31, and backup procedures failed.

VA officials have halted further consolidation moves until they finish evaluating the disruption.

The disruption occurred as VA was moving its health records system, the Veterans Health Information Systems and Technology Architecture (VistA), from local hospital data centers to regional data centers. By creating regional centers, VA officials said they hope to provide better security and standardized management practices to safeguard critical health records and other electronic health information.
VA has two regional data centers serving the West and Northeast regions. Officials planned to open two more regional centers by Dec. 31.

“The root cause of the outage was indeed…human error, more specifically, a failure to adhere to change management procedures.” Jeff Shyshka, Veterans Affairs Department

The VistA disruption affected physicians and patients at VA hospitals, which rely on VistA’s electronic health records for patients’ clinical care. Without VistA, physicians had to revert to paper recordkeeping, which created opportunities for mistakes and slowed the delivery of health services.

“Work to recover the integrity of the medical record will continue for many months since so much information was recorded on paper that day,” said Bryan Volpp, associate chief of staff for clinical informatics at VA’s Northern California Healthcare System.

VA is investigating the incident internally and will hire a company to conduct an independent review to ensure that the department’s contingency plans work, Robert Howard, VA’s chief information
officer, told lawmakers. Howard also is
re-evaluating VA’s regional processing

“We want to examine the whole program and build in more robust backup at the facility level,” Howard said. “We cannot allow hospitals to go down.”

VistA stopped working in northern California because information technology employees did not follow policies and procedures, VA officials said in their initial assessment of the disruption.
“The root cause of the outage was indeed…human error, more specifically, a failure to adhere to change management procedures,” said Jeff Shyshka, deputy assistant secretary of enterprise operations and infrastructure at VA’s CIO Office.

The disruption happened during daytime hours, the busiest time for treating patients at the hospitals. Ben Davoren, director of clinical informatics at VA’s San Francisco Medical Center, related details of the incident during a recent hearing before the House Veterans’ Affairs Committee. He called it “the most significant technological threat to patient safety VA has ever had.”

Shyshka said IT specialists from several VA health care facilities in the region were meeting at the data center. An employee did not follow procedures to vet and implement a system change, he said.
VA resolved the regional data center problem by backing out of a network port configuration change that an employee made earlier that day. That port configuration is the likely cause of the problem, Shyshka said.

“As with any collocation undertaking of this magnitude, there will always be the potential for human error,” Shyshka said. “Ensuring effective communications processes between the teams managing the collocated VistA systems and the IT staff at the local facilities is perhaps the greatest challenge.”

Isolated incident

Charles De Sanno, associate deputy assistant secretary of infrastructure engineering at VA’s CIO office, said he thinks what happened is an isolated incident. “I believe a mistake was made.” De Sanno is one of the architects of the regional data processing center program. He is also executive director of VA’s enterprise infrastructure engineering and northeast operations in New York.
“What you have here is [like] someone crossing the street and not looking and [getting] hit by a car,” De Sanno said. Even if the change the employee made should have been made, it was not authorized, and it was implemented improperly, he said. “We have to ensure that we follow policy and procedure.”

Under VA’s regional processing program, IT systems are physically secured in commercial data centers.

Vendors own and maintain the data center physical plant and network infrastructure and provide support to VA and other customers using the data center.

VA IT professionals operate, manage and maintain the department’s health care information systems remotely, Shyshka said.

About the Author

Mary Mosquera is a reporter for Federal Computer Week.

The 2015 Federal 100

Meet 100 women and men who are doing great things in federal IT.


  • Shutterstock image (by venimo): e-learning concept image, digital content and online webinar icons.

    Can MOOCs make the grade for federal training?

    Massive open online courses can offer specialized IT instruction on a flexible schedule and on the cheap. That may not always mesh with government's preference for structure and certification, however.

  • Shutterstock image (by edel): graduation cap and diploma.

    Cybersecurity: 6 schools with the right stuff

    The federal government craves more cybersecurity professionals. These six schools are helping meet that demand.

  • Rick Holgate

    Holgate to depart ATF

    Former ACT president will take a job with Gartner, follow his spouse to Vienna, Austria.

  • Are VA techies slacking off on Yammer?

    A new IG report cites security and productivity concerns associated with employees' use of the popular online collaboration tool.

  • Shutterstock image: digital fingerprint, cyber crime.

    Exclusive: The OPM breach details you haven't seen

    An official timeline of the Office of Personnel Management breach obtained by FCW pinpoints the hackers’ calibrated extraction of data, and the government's step-by-step response.

  • Stephen Warren

    Deputy CIO Warren exits VA

    The onetime acting CIO at Veterans Affairs will be taking over CIO duties at the Office of the Comptroller of the Currency.

  • Shutterstock image: monitoring factors of healthcare.

    DOD awards massive health records contract

    Leidos, Accenture and Cerner pull off an unexpected win of the multi-billion-dollar Defense Healthcare Management System Modernization contract, beating out the presumptive health-records leader.

  • Sweating the OPM data breach -- Illustration by Dragutin Cvijanovic

    Sweating the stolen data

    Millions of background-check records were compromised, OPM now says. Here's the jaw-dropping range of personal data that was exposed.

  • FCW magazine

    Let's talk about Alliant 2

    The General Services Administration is going to great lengths to gather feedback on its IT services GWAC. Will it make for a better acquisition vehicle?

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above