VA revisits data consolidation plan

Physicians detail clinical system meltdown

When physicians at 17 Veterans Affairs Department hospitals were unable to log on to the Veterans Health Information Systems and Technology Architecture (VistA) and its Computerized
Patient Record System, they discovered that was not their only problem.

  • VA’s regional data processing center in Sacramento, Calif., did not back up the electronic health records system to the Denver regional processing center as it was supposed to.

  • At four medical centers, backup systems were unavailable or overwhelmed. A read-only backup of patient data was unavailable because of a previously planned, periodic updating of a hospital test account.

  • Physicians relied on health summary data stored on several local PCs.

  • Medical employees had to write discharge instructions, medical notes and medication administration records on paper.

— Mary Mosquera

The Veterans Affairs Department must answer new questions about its data center consolidation strategy after its electronic health records system was unusable for nine hours at 17 VA medical facilities Aug. 31, and backup procedures failed.

VA officials have halted further consolidation moves until they finish evaluating the disruption.

The disruption occurred as VA was moving its health records system, the Veterans Health Information Systems and Technology Architecture (VistA), from local hospital data centers to regional data centers. By creating regional centers, VA officials said they hope to provide better security and standardized management practices to safeguard critical health records and other electronic health information.
VA has two regional data centers serving the West and Northeast regions. Officials planned to open two more regional centers by Dec. 31.

“The root cause of the outage was indeed…human error, more specifically, a failure to adhere to change management procedures.” Jeff Shyshka, Veterans Affairs Department

The VistA disruption affected physicians and patients at VA hospitals, which rely on VistA’s electronic health records for patients’ clinical care. Without VistA, physicians had to revert to paper recordkeeping, which created opportunities for mistakes and slowed the delivery of health services.

“Work to recover the integrity of the medical record will continue for many months since so much information was recorded on paper that day,” said Bryan Volpp, associate chief of staff for clinical informatics at VA’s Northern California Healthcare System.

VA is investigating the incident internally and will hire a company to conduct an independent review to ensure that the department’s contingency plans work, Robert Howard, VA’s chief information
officer, told lawmakers. Howard also is
re-evaluating VA’s regional processing

“We want to examine the whole program and build in more robust backup at the facility level,” Howard said. “We cannot allow hospitals to go down.”

VistA stopped working in northern California because information technology employees did not follow policies and procedures, VA officials said in their initial assessment of the disruption.
“The root cause of the outage was indeed…human error, more specifically, a failure to adhere to change management procedures,” said Jeff Shyshka, deputy assistant secretary of enterprise operations and infrastructure at VA’s CIO Office.

The disruption happened during daytime hours, the busiest time for treating patients at the hospitals. Ben Davoren, director of clinical informatics at VA’s San Francisco Medical Center, related details of the incident during a recent hearing before the House Veterans’ Affairs Committee. He called it “the most significant technological threat to patient safety VA has ever had.”

Shyshka said IT specialists from several VA health care facilities in the region were meeting at the data center. An employee did not follow procedures to vet and implement a system change, he said.
VA resolved the regional data center problem by backing out of a network port configuration change that an employee made earlier that day. That port configuration is the likely cause of the problem, Shyshka said.

“As with any collocation undertaking of this magnitude, there will always be the potential for human error,” Shyshka said. “Ensuring effective communications processes between the teams managing the collocated VistA systems and the IT staff at the local facilities is perhaps the greatest challenge.”

Isolated incident

Charles De Sanno, associate deputy assistant secretary of infrastructure engineering at VA’s CIO office, said he thinks what happened is an isolated incident. “I believe a mistake was made.” De Sanno is one of the architects of the regional data processing center program. He is also executive director of VA’s enterprise infrastructure engineering and northeast operations in New York.
“What you have here is [like] someone crossing the street and not looking and [getting] hit by a car,” De Sanno said. Even if the change the employee made should have been made, it was not authorized, and it was implemented improperly, he said. “We have to ensure that we follow policy and procedure.”

Under VA’s regional processing program, IT systems are physically secured in commercial data centers.

Vendors own and maintain the data center physical plant and network infrastructure and provide support to VA and other customers using the data center.

VA IT professionals operate, manage and maintain the department’s health care information systems remotely, Shyshka said.

About the Author

Mary Mosquera is a reporter for Federal Computer Week.


  • Workforce
    Shutterstock image 1658927440 By Deliris masks in office coronavirus covid19

    White House orders federal contractors vaccinated by Dec. 8

    New COVID-19 guidance directs federal contractors and subcontractors to make sure their employees are vaccinated — the latest in a series of new vaccine requirements the White House has been rolling out in recent weeks.

  • FCW Perspectives
    remote workers (elenabsl/

    Post-pandemic IT leadership

    The rush to maximum telework did more than showcase the importance of IT -- it also forced them to rethink their own operations.

Stay Connected