A cost of consolidation?

Despite the push to consolidate data centers in Washington, D.C., and at state capitols nationwide, some would-be centralizers might want to recalculate the risks involved after the recent meltdown of state services in Virginia.

The failure of a single storage system Aug. 25 at a data center near Richmond took down 485 of the state’s 4,800 data servers, knocking out services at three state agencies for more than a week and affecting operations at two dozen others.

State Chief Information Officer Sam Nixon told the Washington Post that the crash was caused by the dual failure of a pair of redundant, 3-year-old memory cards, one of which was supposed to back up the other.

"The thing that is never supposed to happen happened," Nixon said.

Virginia officials said EMC, the company that designed and supplied the storage system, told them the outage had never occurred before in 1 billion hours of system use.

The debacle turns a spotlight on the continuity-of-operations risks associated with consolidating multiple IT operations on fewer, larger data processing and storage systems, as Virginia has done through a $2.4 billion contract it awarded to Northrop Grumman in 2003 and renegotiated this past spring.

The incident also calls into question the design and reliability of supposedly fault-tolerant systems when real-world problems trigger them into action. Information Age reported that a situation similar to the Virginia outage occurred earlier this year when e-mail hosting provider Intermedia lost service to many of its customers after a problem on its EMC storage-area network.

Intermedia officials said a backup storage controller took over when the primary one failed because of a system bug, but the backup device had insufficient capacity to shoulder the entire workload. The company said it has taken corrective action to ensure that there is enough spare capacity on the storage-area network to continue operation in case of future failures.

Virginia CIO Nixon told local TV station NBC12 that he remains committed to centralized IT services despite the recent snafu. But the news will likely have government IT officials elsewhere taking a second look at their system designs and procedures to make sure they can continue operations in case of unexpected problems.

Featured

  • Telecommunications
    Stock photo ID: 658810513 By asharkyu

    GSA extends EIS deadline to 2023

    Agencies are getting up to three more years on existing telecom contracts before having to shift to the $50 billion Enterprise Infrastructure Solutions vehicle.

  • Workforce
    Shutterstock image ID: 569172169 By Zenzen

    OMB looks to retrain feds to fill cyber needs

    The federal government is taking steps to fill high-demand, skills-gap positions in tech by retraining employees already working within agencies without a cyber or IT background.

  • Acquisition
    GSA Headquarters (Photo by Rena Schild/Shutterstock)

    GSA to consolidate multiple award schedules

    The General Services Administration plans to consolidate dozens of its buying schedules across product areas including IT and services to reduce duplication.

Stay Connected

FCW Update

Sign up for our newsletter.

I agree to this site's Privacy Policy.