EPA takes the pain out of patching

Automated approach lets agency spread security patches far and wide .

Like other federal agencies that are trying to avoid being victimized by the next onslaught of computer viruses and worms, the Environmental Protection Agency's Office of Environmental Information is automating efforts to update systems with the latest security patches.

The time between the discovery of vulnerabilities in popular software products and attacks by hackers who create viruses or other malicious programs to exploit those weaknesses continues to shrink. In response, many government information security officers are realizing they need to speed the process of applying fixes to computers enterprisewide. Many of them, including the EPA, are turning to patch management solutions for help.

Cost of inaction

Not long after completing a migration to Microsoft's Windows 2000 software a few years ago, EPA officials began to see a rise in the number of company-issued security updates designed to fix holes that could allow hackers to compromise and, in some cases, take over the upgraded systems.

Unfortunately, applying patches was a labor-intensive process that required the EPA's information technology staff to go from computer to computer to properly install the software fixes, said Bill Sabbagh, security technical monitor at the EPA's headquarters in Washington, D.C. Software updates were complicated because the agency operates numerous regional offices nationwide, whose systems also needed patching.

Even when IT employees thought the job was finished, "it was hard to determine if all [the systems] had been done," Sabbagh said.

Manual patching was costly, but the alternative was even worse. Sabbagh does not have exact figures, but he said failing to keep systems properly patched meant that EPA IT staff would inevitably have to contend with losses of data, time and productivity while fixing infected systems.

The fix

Officials at the EPA's Computer Security Incident Response Capability (CSIRC), based in Research Triangle Park, N.C., evaluated about six patch management products. They wanted a solution that was easy to implement and could be deployed across a variety of computer systems.

For example, the agency operates a variety of operating systems, such as Windows 95, 98, 2000, 2003, NT and XP, Sabbagh said. The solution also had to be scalable to automatically update 24,000 workstations and 1,500 servers nationwide.

They selected PatchLink's PatchLink Update software because it offers both automated patching tools and reporting capabilities. So, for example, if the Slammer worm is infecting systems, the reporting function could tell information security officers at CSIRC if systems in different regional offices are compliant with necessary patches.

EPA officials have loaded the PatchLink agent software on thousands of workstations and servers nationwide, enabling those systems to be part of the automatic patch management system. In addition, they have deployed a PatchLink server and one proxy server at EPA headquarters, a server at CSIRC, and one server each at 10 regional locations.

The EPA's goal is to also install proxy servers for program offices that can accept patches distributed from headquarters. The proxy server could then cache the patches and apply them to other systems in the office at an appropriate time, Sabbagh added.

The payback

Besides eliminating the costly manual process for applying patches, PatchLink offers the EPA "a central point of consolidation for patches," said Chris Andrew, vice president of product management at PatchLink.

EPA officials can retrieve patches from PatchLink's repository, test them against the agency's standard desktop and application configurations and distribute them within minutes across the EPA's network, he said.

Sabbagh declined to say how much the EPA paid for the PatchLink solution. He said other less expensive patch management products are available but added that PatchLink provided the best value for the EPA's needs.

Words to the wise: Doing patch work the right way
  • Don't delay starting. Vulnerabilities in commercial software are increasing, but many can be fixed with patches that have been available for some time. "We're still in the process of deploying, but at least we are doing something," said Bill Sabbagh, security technical monitor at the Environmental Protection Agency.

  • Check before you patch. Test patches on your standard system configurations to make sure they will work with your software before you deploy them. And don’t overlook older systems.