Rainy-day lessons about resiliency

The Internal Revenue Service revamped its continuity-of-operations plans following a damaging flood that shut down its headquarters building in Washington, D.C., for more than five months.

When his beeper went off before dawn June 26 last year, Brian Downs began to realize Washington, D.C., wasn’t experiencing just another summer downpour. Later that day, as floodwaters swamped the Internal Revenue Service’s headquarters at 1111 Constitution Ave., Downs, an IRS operations section chief, met on-site with colleagues. “At that point I knew we were in trouble,” Downs recalled. “I started to put my disaster recovery plan into place, which was to get as much equipment out of the building as quickly as possible.” Before the rains stopped, water rose more than 20 feet in the subbasement, submerging electrical and maintenance equipment. For safety, emergency crews cut electricity to the building.With the help of employees from Apogen Technologies, a network services contractor, Downs and his staff scrambled to keep the IRS’ Office of Chief Counsel’s operations intact. Officials in other departments throughout the headquarters initiated similar continuity-of-operations (COOP) moves to protect the equipment and data associated with 2,200 employees, including the IRS commissioner. “This was a classic case of the head taking the hit,” said Ira Hobbs, who retired as chief information officer at the Treasury Department, the IRS’ parent agency, at the end of 2006. “How does the body continue to function with the dysfunctionality that occurred to the head?”  The answer came in a set of actions — some planned as part of a formal COOP strategy, some improvised on the spot to cope with unexpected challenges. In the end, the flood cost the IRS and the General Services Administration a combined $54 million for cleanup, building repairs, equipment replacement and temporary office rent, and the headquarters remained shuttered for more than five months.An audit by the Treasury Inspector General for Tax Administration (TIGTA) released earlier this year gave the IRS a generally positive review for its response, saying that “due to preparatory and responsive actions, the IRS adequately protected sensitive data.” But the emergency also highlighted some breakdowns that led to adjustments in the IRS’ COOP planning and provided reminders for other agencies that COOP plans must address multiple challenges: How to relaunch critical operations quickly and sustain them in COOP mode for weeks or months.Before the flood, the IRS had a comprehensive COOP plan in place that it practiced regularly each year. One scenario included a headquarters lockdown following a dirty-bomb attack. “But had we planned for a flood of the nature that happened in Washington in June? No, we hadn’t,” said John Dalrymple, deputy commissioner of operations support at the IRS at the time of the flood. He retired shortly before the headquarters building reopened in December and is now a director in Deloitte and Touche USA’s federal practice. IRS CIO Richard Spires declined comment for this story, citing scheduling conflicts.Fortunately, the agency’s practice of housing critical data in a facility in Martinsburg, W.Va., meant the long headquarters shutdown didn’t severely affect work at numerous IRS field offices. “We didn’t have to worry about corporate data, but we had to make arrangements for our core leadership team,” Dalrymple said. The existing contingency plan called for moving the team to offices in the U.S. Mint building, which became command central. “Then we had to decide how we were going to bring all the employees back online as quickly as possible,” Dalrymple said. Many employees were working again within days at temporary offices and some home offices. All staff members had access to IRS computers by late July. But not everything went smoothly, the audit states. For example, in the immediate aftermath of the storm, 104 computers left headquarters before a formal asset-tracking system went live five days after the flood struck, according to the audit. Computer-tracking procedures weren’t originally part of the IRS COOP plan.Although moving the equipment showed initiative, the quick actions left the agency vulnerable to equipment and data losses. The outside-the-lines responses illustrate that COOP plans can’t always anticipate how emergencies play out. “Even though you’ve got a plan and you practice it, there are some tweaks that you have to make based on whatever the incident is and its severity,” Hobbs said. Since the flood, the IRS has incorporated an emergency asset- tracking form and related training into its COOP plan.IRS officials also have a renewed interest in expanded telework operations. During the headquarters shutdown, the agency worked with AT&T to establish a secure virtual private network to enable some employees to work from home offices. However, the TIGTA audit recommended that the IRS do more to develop a telework business case to help promote greater reliance on telework among department managers. Lack of a more extensive telework infrastructure created hardships on IRS employees during the building closure, according to the audit, which reported that 1,700 staff members — almost 80 percent of the headquarters’ workforce — were on administrative leave in the week after the flood. The audit opinion bolsters other research, including a study done for GSA by Booz Allen Hamilton that estimated a hypothetical $15.6 million telework investment at a 50,000-person agency could yield $31.1 million in potential cost benefits.But for telework to be effective in situations such as the IRS flood, security must be a prime consideration, said Dave Jerome, a principal at Booz Allen Hamilton. “Telework isn’t a matter of just having enough bandwidth to connect everybody. You’ve got to build in security from the very beginning,” he said. “It’s much easier and cheaper to [establish] the proper security protocols upfront than it is to develop a system and then have security thrown on top of it.”  Part of telework design requires deciding which jobs are best performed from home offices and which are best accommodated by grouping people at satellite offices, he added. The advantages of common telework hotels are greater control of the work environment, including security technology and procedures, and access to office resources, such as conference rooms.In response to the audit’s recommendations, IRS officials said they plan to “advocate the consideration and use of telecommuting” for COOP. Current and former IRS employees said they learned other lessons beyond what the audit outlined. Downs developed a deeper appreciation for the collaboration necessary between the information technology and mission staff members for successful COOP planning. “It was obvious during this particular situation that the [COOP plans] weren’t closely tied together,” Downs said. “If it wasn’t for us IT people knowing the [chief counsel] business as well as we did, I don’t believe we would have pulled this off.”Hobbs said that close coordination should include commercial service providers and partners. “You have to reach out across government and industry to make sure [COOP] happens in a smooth way that is transparent to the users and clients of the organization,” he said.Dalrymple said the IRS could have done more to ensure consistent data backup procedures. “In some instances, people hadn’t backed things up on local servers as frequently as they should have. Things like that caused some blips,” he said. “Now the IRS has an ability to back those [resources] up corporately on a regular basis.”The emergency also reinforced the need for adequate system redundancy and procedures to quickly make the failover technology available to staff members. However, high availability can come at a steep financial price, said Shawn McCarthy, director of research for government vendor programs at Government Insights. “A lot of agencies have plans. Whether they have the funding and the manpower to enact those plans is a different situation,” he said. To balance high availability and budgets, Jerome recommended that agencies prioritize critical functions on a continuum that ranges from resources that must never go down to those that can be idle for hours or days without harming the agency. Focus spending on “what you are required by law to provide under any type of circumstances,” Jerome said. Then protect top priorities with expensive redundant backup technology with immediate failover capabilities, he said. Finally, agencies need to spend more time testing their existing COOP plans to guard against surprises in the midst of a real emergency, Jerome said. “Constantly make sure that people understand their responsibilities. Each time you test the plan you are going to find things that didn’t work exactly the way that you thought they would. COOP is a living plan that has to be updated on a periodic basis.”

Treasury Inspector General For Tax Administration’s report

Related Links

















Tracking assets














Focus on telework












New lessons














Joch is a business and technology writer based in New England. He can be reached atajoch@worldpath.com
No reprieve from taxesFor those who might have secretly hoped the five-month disruption of operations at the Internal Revenue Service’s headquarters would have delayed the April 17 income tax filing deadline, here’s why there is no reason to celebrate.

  • Tax data was stored in Martinsburg, W.Va., far away from  flooded areas in Washington, D.C.
  • Employees followed continuity-of-operations procedures to quickly create a command center for senior executives in offices at the U.S. Mint.
  • Physical transfers of computers from headquarters to alternative sites began immediately after the building was deemed temporarily uninhabitable.
  • All dislocated staff members gained access to IRS computers within a month after the flood.
  • A secure virtual private network enabled some of the IRS workforce to have home-office access to data and applications.

— Alan Joch

A lesson from the IRS: Plan to improviseWhen Brian Downs, an Internal Revenue Service operations section chief, and his staff and contractor Apogen Technologies launched the continuity-of-operations plan for the Office of Chief Counsel in June 2006, they focused on eight Microsoft Exchange e-mail servers. In addition to providing essential communications among counsel staff members, the e-mail system acts as a dynamic repository for collecting and sharing information about ongoing litigation. The data on those servers and in a storage-area network was key to maintaining operations after the headquarters building became uninhabitable.

But with no power to run the computers, rescuing information from fourth-floor offices was a challenge. “There was no way we were going to be able to replicate that information to a different site quickly,” Downs said.

So the group devised a manual workaround. They planned to hand-carry the 80-pound servers down the dark stairwells using flashlights to find their way. “It had to be 110 degrees in that building, and there was an electrical smell,” Downs recalled. “You could tell there was definitely something that had burned in the basement.”

Before removing the equipment, the team needed to copy about 4T of data from the storage-area network to local disks on the servers. To return power to the fourth-floor counsel offices, Apogen employees fashioned a cable out of 12-gauge wire. They ran the cable from the computer room, down a stairwell, to rented generators in the parking lot. Downs used another generator purchased from a retail home center to power 120-volt equipment.

With temporary power flowing, the technicians used a standard Move command in the Exchange software to transfer data from the SAN. “Fortunately, we had enough drive space on the local servers,” Downs said.

Using wireless local-area network connections on the servers, the group also moved data from another application to 500G external hard drives, which they purchased at a local electronics store.  They then loaded the units into trucks and delivered them to nearby IRS offices at 950 L’Enfant Plaza, which became the new operations base. The first server came back online by the next afternoon, and the counsel’s entire e-mail system was functioning later that evening. “Through this process, we didn’t lose one e-mail message,” Downs said.

An audit of the IRS’ response to the June emergency and its aftermath criticized the agency’s initial lack of an asset-tracking system to account for more than 100 computers as they left the headquarters. Although some counsel equipment was removed before the IRS initiated formal tracking, Downs said all the devices were accounted for.

“When we removed the equipment out of 1111 [Constitution Ave., the IRS’ headquarters], we had somebody from our project management staff scan all the incoming equipment at 950 L’Enfant Plaza to ensure an accurate account of all equipment assets,” he said. “The same procedures we have always used in our organization for tracking assets are still in place. We were one of the few who didn’t have to modify our procedures.”

— Alan Joch
X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.