Dial VOIP in case of emergency

Voice-over-IP technology finds a new fit with business continuity strategies

When officials in Orange County, Fla., installed a voice-over-IP phone system in 2003, they quickly achieved the goal of cutting telecommunications costs by as much as 30 percent. But after a series of hurricanes hit a year later, county managers discovered another reason to love the technology.

To provide services for citizens affected by Hurricane Charley and two other storms in 2004, the host county for Disney World created a temporary emergency center in an empty warehouse that the county was planning to renovate. “It had nothing in it, including [no] communications infrastructure,” said John Amiot, enterprise operations manager for the county.

So officials trucked in PCs and network gear that supported VOIP, and within 24 hours, the facility opened for business with a line of storm-weary citizens already forming at the door.

“Our people could cut checks and provide all the things that people needed in order to survive,” Amiot said. “We were able to deploy in 24 hours because we had voice-over-IP technology already in place.”

Since then, the county has integrated IP telephony into its formal continuity-of-operations (COOP) plans by installing it in a regional computing center.

Expanding role
Business continuity plans are increasingly relying on new technologies to keep organizations running in the aftermath of natural disasters, terrorist attacks and more mundane incidents, such as local power outages.

“IP telephony and continuity of operations are an excellent marriage,” said Jim Biskaduros, a client delivery executive who specializes in security and intelligence at systems integrator EDS. In Orange County, the newly wired warehouse became an extension of its infrastructure. An IP network provided connectivity among employees and the county’s main offices.

“It was as if we had just opened a new building,” Amiot said. “This was in the center of one of the most devastated areas in the county, and it was strategically advantageous for everybody to come there versus getting them downtown while trees were down everywhere.”

By the time the last of the storms plowed through the area, the warehouse had logged more than 226,000 calls. The system automatically routed some calls normally intended for headquarters to the site.

Other agencies are also discovering IP telephony’s flexibility. The Department of Veterans Affairs turned to IP-enabled mobile communications trucks to keep Gulf Coast hospitals running after Hurricane Katrina hit last year. After finding success, the agency is extending contracts to formalize those backup communications tools and COOP plans.

Similarly, the Education Department didn’t initially have COOP in mind when it installed IP telephony in 2002. But the technology’s reliability spurred new ideas for how the department could use it.

“We said, ‘Hey, what if?’” said Peter Tseronis, Education’s director of network services. “We went back to the drawing board to seriously consider COOP.”

At those and other agencies, IP telephony provides resilience in emergencies because the networks that keep voice traffic flowing consist of widely distributed servers and connections that can pick up the slack if individual segments quit working. Traditional phone networks depend on point-to-point connections with fewer secondary options. Also, emergency workers can create ad hoc IP voice connections via IP phones or laptop computers with soft phone software that link to data network connections, including DSL and cable modems.

Upfront planning
Nevertheless, using IP telephony for COOP is still a relatively rare application at most agencies. “IP telephony is on their radar screen, but I wouldn’t say in general they’ve made lots of progress,” said Mike Corrigan, lead consultant at Suss Consulting. That’s partly because significant groundwork is necessary to make IP telephony reliable enough for COOP.

The first step for any basic IP telephony installation is for IT administrators to secure the data network that supports voice communications, said John Speicher, market development manager at Cisco Systems. That includes installing redundant IP exchanges — the computer servers that provide voice capabilities — network switches and uninterruptible power supplies that support the hardware.

COOP planners also need to understand the performance characteristics of their voice systems. “When you’re building your [request for proposals], account for survivability,” said Nora Freedman, research analyst for enterprise networking at market researcher IDC. Specifically, ask if solutions support automatic failover, and if so, how quickly, Freedman added. Failover means the systems smoothly and automatically reroute their functions if one or more component ceases to work properly.

Security is a greater concern for IP telephony than in traditional public switched telephone network connections. Traditional “voice is hard-wired from phone to communications closet to switch,” said Guy Clinch, solutions director for government and education at Avaya. “To compromise calls, you have to gain physical access. The data world is a lot more distributed with more access points.”

Experts say security needs to be an upfront consideration for COOP plans using IP telephony, not something that organizations cobble together in the middle of an emergency.

Agencies concerned about reliability and security, including those in the defense and intelligence communities, need private network links among the main headquarters and COOP sites. “I don’t think at this time the public Internet is reliable enough for a COOP-type service,” Biskaduros said. Another security strategy is to separate voice and data transmissions using Multiprotocol Label Switching and virtual local-area network technology. The former uses management software to string particular computers together into groups on the network rather than relying on hard-wired connections for the segmentation. The virtual approach simplifies network reconfigurations in an emergency.

Once administrators create individual voice streams, they can secure them with a virtual private network that encrypts communications. Separating data streams also delivers quality-of-service benefits. Administrators can give voice traffic priority over data packets to eliminate dropped or unintelligible calls.

COOP strategies that dispatch critical workers to home offices should provide them with VPN software to protect communications via the Internet, said Siafa Sherman, vice president of systems engineering at Nortel Government Solutions, an IP equipment vendor.

In addition to security considerations, the VA took an added step to prepare Gulf Coast facilities for the 2006 hurricane season. The department negotiated with telecom companies to ensure that all the services in the agency’s wide-area data network are part of the priority service restoration program that such companies make available to federal agencies.

“We want to make certain that we are queued up as higher priorities where life surety issues come into play,” said David Cheplick, deputy director of the VA’s Office of Telecommunications.

Service component
After its first IP telephony building came online in late 2002, Education used the technology to establish voice failover capabilities.

“The way we’ve structured and configured our IP communications network, we don’t have a single point of failure with our IP PBXs,” Tseronis said.

Education’s IP exchanges cover four sites in Washington, D.C. If an IP PBX fails in one of the buildings, other servers in the immediate cluster of computers automatically take over the traffic to keep communications running. If an event disabled the entire D.C. area communications infrastructure, calls would flow to Education’s disaster recovery site in Georgia, which would act as a communications hub to connect satellite offices.

The agency also installed routers that can send calls onto the public switched telephone network if the wide-area data network crashes. “That’s a definite no-brainer best practice,” Tseronis said. But he pointed out that such automatic failure capabilities take more than just buying hardware and expecting it all to automatically work. Like their VA counterparts, Education officials discussed their COOP plan with telecom service providers, which are responsible for rerouting calls if network segments go down.

“There is some manual intervention to ensure that if something fails, an alternative kicks in,” Tseronis said. “In some cases, it can be automated, but it takes working with the telcos.”

Discussions with service providers should occur early in the COOP planning process. “If you wait until the last minute, after you’ve developed your bill of materials to buy a new solution, it is not going to be well thought out,” Tseronis said. “That relationship needs to be born early on and continually cultivated.”

Joch is a business and technology writer based in New England. He can be reached atajoch@worldpath.com.

Rolling communications

No matter how much work goes into building resiliency into a communications network, exceptional forces can sometimes bring it down. The Department of Veterans Affairs confronted this fact last year when Hurricane Katrina severed communications with 10 medical centers stretching from the Florida panhandle to New Orleans.

“The lesson we learned was that [wide-area network] service was restored several days before reliable voice service,” said David Cheplick, deputy director of the VA’s Office of Telecommunications.

The only answer in the immediate aftermath came from mobile communications vehicles equipped with dishes that established IP telephony connections via Earth-orbiting satellites. “We connected both voice over IP as well as data links, and that proved to be a very good solution for interim communications at our outpatient clinics and major medical centers in that area,” he said.

The satellite solution wasn’t part of the existing continuity-of-operations plan last year, but the VA now has reserved mobile satellite units and services for its Gulf Coast and southern Atlantic facilities. Monthly costs for the backup services are about $1,000 for each of the six operations in the areas.

Communications hardware vendors are tapping into this growing interest in mobile communications units. Avaya recently announced two versions of mobile units that use IP technology to create rolling network gateways. The price is $25,000 to $75,000, depending on the sophistication level.

Getting help to VOIP callers

The flexibility of IP telephony can be a blessing and a curse. It can keep communications flowing when traditional phone lines are down, but it can also make it difficult for first responders to find the source of 911 calls if agency employees move IP phones to alternate offices or sign on with voice-enabled laptop computers.

To help, networking vendors and service providers are offering proprietary location identification services. For example, Avaya sells software that prompts callers to enter a local phone number if, for example, they sign on to the voice network from home rather than from the main office number. A database maintained by the public safety infrastructure associates a physical address with the number. Similarly, Cisco Systems sells technology that identifies a phone’s location, including the particular floor and office, within a Cisco infrastructure.

A company called eTelemetry has introduced a network device that crosses platform boundaries. Called Locate 911, the hardware and software bundle prompts callers to identify themselves when they log on to the network so the system can match them to the physical connection on the network.

The appliance is network- and phone-agnostic, said Alan Schunemann, eTelemetry’s chief technology officer. The product also gives network managers the option of locking IP phones to a specific switch port to avoid potential location problems and prevent employees from moving the phones.

2014 Rising Star Awards

Help us find the next generation of leaders in federal IT.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above