CIOs preach the power of resilience

Rear Adm. Robert Day, USCG

Rear Adm. Robert Day, the Coast Guard's Cyber Command commander and CIO, stresses that agencies must physically rehearse their contingency plans -- repeatedly.

If federal IT managers have practiced their disaster recovery plans and their staffs know how to improvise, agencies may not be completely crippled by the potentially devastating cyber "Pearl Harbor" that former Secretary of Defense Leon Panetta ominously predicts could be lurking.

That was the message several current federal CIOs conveyed at a March 11 panel discussion on continuity of operations at Symantec's Government Symposium. In a keynote speech before the panel discussion, Panetta reiterated his October "Cyber Pearl Harbor" warning in light of growing tensions with Russia, competition with China and a restive world stage.

In his symposium speech, Panetta dubbed cyberwar "the most serious 21st century threat." Hurricane Sandy, he said, demonstrated some of the ruinous effects on infrastructure that a cyberattack might inflict.

In the face of such danger, federal agencies not only must have a disaster plan, but they should physically rehearse it -- repeatedly, said Rear Adm. Robert Day, the Coast Guard's Cyber Command commander and CIO.

Without practice, he said, plans rot on the shelf or fail in the face of real-life events. With increasing cybercrime, natural disasters and other potentially catastrophic events in the wings, not to mention local tech snags and outages, Day said the Internet and networks are more imperiled than ever. Even without a dramatic event, he said, federal agencies' network facilities eventually "will go down" because of some weakness, such as a router failure or other snafu.

In the end, natural disaster or cyberattack results are the same, Day said, with network facilities that don't work. It's then that an agency's plans had better be up to date and well-rehearsed. "We see so few organizations doing that."

Co-panelists Wolfe Tombe, chief technology officer at Customs and Border Protection, and John Dobriansky, executive advisor at the Federal Aviation Administration, agreed. All three noted substantial network outages at their agencies in the last few years.

Tombe said a local CBP router network outage at Los Angeles International Airport a few years ago that lasted an hour or so, causing a 17,000 passenger back-up at security lines, as well as air traffic delays because passenger manifest lists could not be checked. CBP officers, he said, improvised and used their cell phones to exchange information. Day noted Hurricane Sandy made the Coast Guard use its fallback servers in West Virginia linked to small aperture terminals set up in the disaster zone to continue emergency radio communications in the stricken mid-Atlantic area after the storm surge shorted out the telephone network. And Dobriansky said a 2012 fire at an FAA facility in New Jersey forced the agency to transfer some of its air traffic control to another facility in Melbourne, Fla. The outage lasted two weeks, he said.

In those instances, the disruption showed not only the weaknesses but also the strengths of their network disaster recovery plans. Day noted the importance of lower-tech workarounds that might be overlooked in such situations, especially by younger workers familiar only with the online world. "Young workers won't know how to do it," he said. "It makes me stamp my feet. Younger workers don't know what a fax machine is."

Day also noted that younger workers can also have ideas that older workers might overlook. After the earthquake that shook the middle-Atlantic states and Washington, D.C., in 2011, younger federal workers milling around on the mall in the aftershocks texted their colleagues rather than trying voice. "They knew that data was working and voice wasn’t," he said.

Day and Tombe said they perform regional and local drills of disaster recovery plans regularly. In Day's case, those exercises come just before hurricane season. "We do a data center fail over, forcing things onto our disaster recovery site. At the local end we do a once-a-year drill. Each sector physically re-establishes itself at its disaster recovery site." That means people physically pick up their gear and move hundreds, or even thousands, of miles to the alternate site.

Day recommended that IT managers file their most critical data and information on a disk or two and put those disks in their "go-bags."

"I can't tell you how many times I've heard 'I didn't think of that,'" he said.

About the Author

Mark Rockwell is a staff writer covering acquisition, procurement and homeland security. Contact him at or follow him on Twitter at @MRockwell4.

The 2015 Federal 100

Meet 100 women and men who are doing great things in federal IT.


  • Shutterstock image (by venimo): e-learning concept image, digital content and online webinar icons.

    Can MOOCs make the grade for federal training?

    Massive open online courses can offer specialized IT instruction on a flexible schedule and on the cheap. That may not always mesh with government's preference for structure and certification, however.

  • Shutterstock image (by edel): graduation cap and diploma.

    Cybersecurity: 6 schools with the right stuff

    The federal government craves more cybersecurity professionals. These six schools are helping meet that demand.

  • Rick Holgate

    Holgate to depart ATF

    Former ACT president will take a job with Gartner, follow his spouse to Vienna, Austria.

  • Are VA techies slacking off on Yammer?

    A new IG report cites security and productivity concerns associated with employees' use of the popular online collaboration tool.

  • Shutterstock image: digital fingerprint, cyber crime.

    Exclusive: The OPM breach details you haven't seen

    An official timeline of the Office of Personnel Management breach obtained by FCW pinpoints the hackers’ calibrated extraction of data, and the government's step-by-step response.

  • Stephen Warren

    Deputy CIO Warren exits VA

    The onetime acting CIO at Veterans Affairs will be taking over CIO duties at the Office of the Comptroller of the Currency.

  • Shutterstock image: monitoring factors of healthcare.

    DOD awards massive health records contract

    Leidos, Accenture and Cerner pull off an unexpected win of the multi-billion-dollar Defense Healthcare Management System Modernization contract, beating out the presumptive health-records leader.

  • Sweating the OPM data breach -- Illustration by Dragutin Cvijanovic

    Sweating the stolen data

    Millions of background-check records were compromised, OPM now says. Here's the jaw-dropping range of personal data that was exposed.

  • FCW magazine

    Let's talk about Alliant 2

    The General Services Administration is going to great lengths to gather feedback on its IT services GWAC. Will it make for a better acquisition vehicle?

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above