Y2K Contingency Planning: What If...
When it comes to disasters, cities and counties throughout the United States are prepared to face just about anything: hurricanes, tornadoes, blizzards, earthquakes, mudslides-even terrorist attacks and nuclear war. But the crisis that many people expect to occur when the internal clocks on computer systems roll forward to Jan. 1, 2000, has city managers and emergency management teams more than a bit anxious.
After all, no matter how much software remediation and testing is done, there remains the risk that comes with having to rely on unknown and untested external systems for vital services such as power, communications and water. And then there's the matter of embedded chip failures in such disparate tools as car engines, telephone switches, power grid systems, water treatment systems and hospital equipment.
"The difference with this [crisis] is that we know exactly when it will occur, but we have no idea how bad it's going to be," said Bob Cass, city manager of Lubbock, Texas. "There are some people who say it's much ado about nothing and others who swear up and down that this is Armageddon in the making. Personally, I think there will be some failures and a few inconveniences but nothing that can't be fixed fairly easily. But since we are responsible for the health, safety and welfare of 200,000 people, we're going to prepare for the maximum possibility."
With only a year left, such uncertainty about whether mission-critical systems will hold up under the impending date change is forcing city and county managers to begin developing extensive contingency plans that rely on more old-fashioned systems, such as fuel-powered generators, two-way radios and even manual typewriters, to ensure that life goes on when the clock strikes midnight.
"The problem is that you can have every single one of your systems compliant and tested, and there are still no guarantees," said Garvin Brakel, chief information officer of Spokane, Wash. "There's just too much interdependence between systems, and there's no real way of telling where there might be a breakdown in the system."
Manual backup plans are being touted to local governments with a growing fervor by major civic associations, state officials and Congress. The federal Year 2000 czar, John Koskinen, summed up the situation this summer in a speech at the annual meeting of the National Association of Counties (NaCO). He said the ultimate burden for Year 2000 failures would fall on the shoulders of local government. If and when technology defaults start cutting off vital services, he warned: "The people with the pitchforks will come looking for you, not me."
With such motivation in mind, many localities already have begun contingency planning in earnest, including Contra Costa County, Calif.; Los Angeles; Spokane, Wash.; Denver; Prince William County, Va.; and Montgomery County, Md.
But one locale is well ahead of the pack. Lubbock, a veteran of emergency management that has dealt mostly with killer tornadoes, has developed a game plan that includes double- and triple-staffing police, fire and rescue personnel, low-tech diesel generators and even cutting off its self-owned electric system from the national grid. In a move last October that grabbed everyone's attention, Lubbock put its fledgling Year 2000 contingency plan to the test, simulating a series of scenarios including power and natural-gas outages, a hospital fire, a riot at a grocery store, price gouging and a murder-all in the middle of a simulated ice storm and blizzard.
"We found some areas of vulnerability, but we're going to keep working on it over the next year, testing other scenarios," Cass said. "It's an ongoing effort, but we'll be ready and in good shape by Dec. 31, 1999."
A Community Effort
The trend toward contingency planning, observers said, finally takes the millennium bug out of the domain of technology managers and turns it over to the community at large. Not only are traditional emergency management personnel such as police and fire crews being tapped for duty, but city and county managers are rounding up the help of utility companies, fuel suppliers, hospitals, social services, private businesses and even libraries and churches.
"The Y2K crisis has never been a technology problem per se," said Michael Humphrey, business director for telecommunications and information at Public Technology Inc., the technology arm of the National League of Cities and NaCO. "It's always been a management problem; [it is] a community problem that needs to be dealt with on that level."
Ultimately, communities that will fare best will be those that possess the characteristics that enable survival during any major crisis: a solid communications infrastructure, cooperation between public- and private-sector officials, a citizen base that is aware and psychologically prepared, a strong sense of priority about the health and safety of citizens, and a contingency plan that is well-rehearsed and easily implemented.
But this doesn't mean that technology is suddenly excluded from the community. Chief information officers and their local technology staffs have a critical role to play, although their main duty is plain: "My responsibility is to ensure that the technology that I have control over is working. It's that simple," said Steven Steinbrecher, CIO for Contra Costa County, California's sixth-largest county, with more than 1 million residents.
On the other hand, the identification of mission-critical systems and embedded chips, and the continual remediation, testing and monitoring of systems, equipment and interfaces offers residual value to emergency management teams: the ability to point out areas where failures are likely to occur together with the ability to better align priorities.
"If you are constantly in touch with your suppliers and others with systems that you depend on, you may not be able to guarantee with any kind of certainty that their system is going to stay up and running, but you will definitely know if they are having problems bringing their systems up to Y2K standards," said Masood Noorbakhsh, CIO for Northern Virginia's Prince William County. Noorbakhsh reports regularly to the county board of directors and is in constant touch with the head of emergency management. "In that instance, you'll be able to predict that there will be a likely breakdown in that area and [you can] make sure that a solid backup plan has been put in place to compensate for it."
Such information communicated between CIOs and emergency management personnel is critical to developing an effective contingency plan for the Year 2000 crisis because while most cities already have disaster plans in place, only a few of the basic elements can be applied.
"Typically, when we have a natural disaster, it does not radically affect our technology," said Tim Cuthriell, director of the Denver Office of Emergency Management and the captain of the police department. "You might lose power, but you can usually communicate with personnel on the streets and get information out to the public. With this one, we just don't know what we're going to be dealing with, and we have to plan for every possibility."
Denver, like many other communities, already has scheduled several simulated, or tabletop, exercises over the next year to help fine-tune its ability to work without vital services such as communications, power and information systems. In these critical tests, technology staff can again provide a valuable service by developing realistic scenarios.
"A police chief or a water department official may know what his mission is and what has to be done, but they may not have a real sense of what might fail or hypothesize how those failures will play out," Spokane's Brakel said. "A technology person, on the other hand, can do that pretty easily."
Cass said technology members did, in fact, play a key role in adding reality to the Lubbock dry run. A key component of any test, he noted, is the control team. "You want to have a highly competent control team, not just operationally but technologically, that can set up the exercise and devise the scenarios while keeping the normal group completely in the dark about what's going on," he said. "That independence ensures that the emergency team is not over-prepared and ready with all the right answers. You want them to be surprised because that's how it could be during the real event."
Taking the Lead
Technology personnel can and should take up the call to educate government officials and disaster personnel on the gravity of the Year 2000 crisis as well as answer the need to develop contingency plans-even when remediation and testing is being performed, industry observers said.
Awareness of the overall problem, Steinbrecher said, still remains too low for comfort. A recent survey by the California Association of Local and State CIOs (CALSCIO) bears that out. Among its findings: More than 80 percent of the entities questioned had no contingency plan in the works, and more than 42 percent of those respondents stated that they weren't ever going to develop one.
"I don't think there's a real awareness out there that you really need to be looking at contingency plans," said Jon Fullinwider, CIO of Los Angeles County and a board member of CALSCIO. "When it's really going to hit people is when they wake up to the fact that they've got a problem with their Y2K compliance and then realize that they don't have sufficient time to deal with it. At that point, they'll begin to look at their next option, which is the contingency plan. And the question is, when they come to that realization, are they going to have time to implement an effective plan?"
In cases where there is little executive commitment, it may be up to the CIO to make city and county managers aware of the need for implementing backup solutions. "It's really a situation where you're going to have to be proactive," Brakel said, noting that the Internet is full of various takes on contingency planning. "Let's face it, there are a lot of people out there who are in denial about this issue."
The need to somehow make the public aware without panicking them is another job of CIOs. Bringing the general public into the equation has always been a sticky subject for local, state and federal officials who worry about a "Chicken Little" scenario, where too much concern expressed on their part could cause obsessive stockpiling of goods, widespread withdrawals from financial markets and bank accounts, and other anti-community behavior.
"If, for example, everybody goes in on the last week of December and tries to fill their gas tanks or take money out of the bank, there isn't going to be enough to go around," Fullinwider said. "The supply system couldn't deal with an all-out assault like that under any circumstances. So you do have to be careful what you tell people because you end up with a manifestation of the very kind of problem that you're trying to guard against."
Most officials do believe, however, that the public needs to be informed of what's going on. "Citizens have a right to know where we stand on all of this so they can make their own personal decisions as well," said Don Heiman, the CIO of Kansas who noted that he instructs local officials to make plans based on their dual roles as private citizens and community leaders. "We believe that when people have good information, they have the opportunity to make calm, rational decisions and are less likely get into this panicky, hoarding behavior."
Technology officials can aid the effort by using the World Wide Web and other resources to detail their remediation and testing efforts and let residents know about the manual backup plans that are being put into place. In short, they can get the point across that communities are not attempting to swing into the millennium without a safety net. "It's a very fine line that you have to walk, but we feel we're going to be better off if we try and educate people about the problem and what we're doing to alleviate it," Noorbakhsh said. "I don't think you're serving anybody's interest by keeping people in the dark on this."
Lubbock has gone so far as to include a Tips for Preparing section on its Year 2000 Web site (y2k.ci.lubbock.tx.us) that outlines the personal contingencies advocated by the Cassandra Project, a grass-roots organization concerned with community preparedness. Among its suggestions: Set aside cash throughout the year (enough to pay bills for two or three months), have a portable radio and plenty of batteries on hand for information, stock up slowly on canned goods and other pre-packaged foods, get hard copies of your medical records before the date change, obtain an alternative heat source that doesn't require electricity and have plenty of candles, flashlights and other battery-powered light sources on hand.
"We're fairly familiar with dealing with natural disasters, and we had a major regional electrical outage a few years back," Cass said. "So our people are fairly familiar with the need to be prepared for anything. That certainly helps us out."
Fullinwider notes that no matter what information is relayed to the public, it needs to be coordinated through a few controlled sources. "This is not the time for some program manager to stand up and talk about what he or she knows or what they think is going to happen," he said. "You need to have somebody already on board that's part of the process [so] that you're giving out consistent information."
After the Fact
U.S. cities will have the added benefit of knowing what's coming their way, thanks to the fact that Asia's clocks will turn forward 14 to 17 hours before the change hits American time zones. That will give communities a chance to adjust contingency plans and possibly tinker with computer programs. But in the aftermath of the date change and in the face of computer and chip failures, CIOs and other technology staff must step up to the plate and provide instructions on how to proceed.
In fact, getting back on track needs to be part of any contingency plan. "If you don't have something pre-defined, like [a time frame of] four hours or 24 hours, then you begin rationalizing," Fullinwider said. "Otherwise, you'll have people in there working on the system, stating: 'Give me another hour and I'll have it fixed.' Meanwhile, we're 10 hours beyond when we said we'd invoke the plan. So whatever metric you decide on, it needs to be determined beforehand. That way, if a breakdown occurs, there's no decision; you just invoke the plan."
At that breaking point, CIOs must coordinate efforts to find and fix problems. Many observers recommend lining up extra technology staff now or even hiring an outside contractor. Kansas, for example, has contracted with a disaster-recovery firm, SunGard Data Systems Inc., to help its state agencies and localities recuperate as quickly as possible.
Among the benefits of such an arrangement is the ability to transfer, or piggyback, many IT functions to SunGard's data processing center in Philadelphia. However, to do that effectively, Heiman said, communities have to begin aligning their infrastructures with SunGard's now.
The contingency plan still has relevance even beyond the date change. For example, it can and should act as a blueprint for the recovery effort of the systems that were identified in the plan as top priorities should be the first to be addressed in the event of a breakdown.
Costa Contra County, for example, has already outlined its triage list, with four levels of severity assigned to every system. Anything related to a function that is necessary to life, health, safety and welfare ranks as "Sev 1." Any function that results in monetary loss to the county gets a "Sev 2" designation, including payroll and property taxes. "We miss two property tax rolls, and we're dog meat," Steinbrecher said. "So obviously that's one that we're concentrating on now. But if it fails next year, we'll be putting our people on it right away."
Most CIOs note that they hope all this preparation and concern will be for naught. "I have no choice but to be concerned about this, but it would be great if come February, everyone can sit here, recall all the crazy worrying that I did and have a good laugh at my expense," Steinbrecher said. "That would make me a really happy man."
-- Heather Hayes is a free-lance writer based in Stuarts Draft, Va.
1. California Year 2000 Readiness
* 73.6 percent of jurisdictions had a Year 2000 compliance plan.
* 58.1 percent had no budget for their plan.
* 82 percent expected to be Year 2000-compliant by the third quarter of 1999.
* 38.2 percent believed they did not have or use any embedded chip systems.
* 80.9 percent said they have no contingency plan in place.
Source: California Statewide Intergovernmental Year 2000 Task Force's survey of 466 cities, 58 counties and a sample of 7,000 districts in the state.
2. Kansas Contingencies
Kansas has produced a "Guidebook 2000," which features guidance on contingency planning developed by Keane Inc., an information technology consulting firm based in Boston. Some examples of guidance include:
Service: Municipal and public utilities and the power grid.Area of concern: Loss of power that could disrupt heating, lighting, communications and other amenities of daily life.Contingency plan: To secure and activate manual stand-by generators. Top-off fuel tanks in December 1999 and procure additional supplies if necessary.
Service: Lock-upsArea of concern: Prison escapesContingency plan: Perform lock-downs manually. Disable any computerized lock-down controls.
Service: Traffic controlArea of concern: Traffic light malfunctionContingency plan: Use police overtime or put together an auxiliary police force to manually direct traffic.
Service: Electronic payroll depositArea of concern: Employee payments may be late or fail entirelyContingency plan: Print out the last payroll in December 1999 and write checks manually.
3. Montgomery County, Md.
The Montgomery County Year 2000 Project Office has developed specific Year 2000 contingency plan preparation guidelines. Step-by-step instructions for successfully keeping services up and running include:
* Assign contingency plan and recovery team members and designate a lead point of contact.
* Perform risk analysis/triage.
* Review triage results and determine priorities.
* Develop and finalize recovery strategies.
* Develop implementation and resource requirements.
* Develop and document procedures.
* Train recovery team members.
* Test and evaluate contingency plan.
* Certify and approve final contingency plan.
* Update and revise the plan.
* Report progress and issues.
* Monitor regional preparedness issues, including utilities.