New Mexico IT staff learns the hard way about what happens when security tools aren't in place
Frank Woods remembers his exact whereabouts when he found out just how painful
and costly the words "I love you" could be.
It was May 4, recalled Wood, chief operating officer of the New Mexico State
Highway Department. He was in a meeting in Albuquerque when he received
a priority call from his Santa Fe office. The news struck him like a freight
train: Massive amounts of IT infrastructure in his department had been infected
by the mercurial "love bug" e-mail virus, and it was spreading a file- and
"We use a lot of JPEG files, digital cameras, and scanned photographs in
our engineering design process," Wood said. "Our hardest-hit area was engineering.
We believe there may have been 50,000 to 60,000 files affected."
The task of cleaning up the infected machines was hampered by a tricky wild
card: The department was in the midst of integrating Microsoft Corp. Exchange
and Microsoft Outlook as its single e-mail program and installing a complete
Tivoli Systems Inc. framework infrastructure. Making matters worse, Wood
said, his organization did not have an emergency plan in place to deal with
The first order of damage control was to shut down every piece of equipment
throughout the legacy system. It lay dormant for about two days.
"It's safe to say everything from design processes to having systems and
communications to our consultants and contractors — every part of our technology
was shut down. We literally went back to paper and pencils for many of our
processes," Wood said.
The second step was to set up an IT crisis management team; individuals
from Wood's staff, Tivoli, Compaq Computer Corp., Fiori Industries, and
the Foundation Consulting Group issued hourly reports.
Less than 24 hours after the initial lock down, individual parts of the
system were turned back on, only to be aborted after it became apparent
that the worm continued to propagate through outside sources. Because the
Tivoli environment was not yet fully implemented, any remote user not accounted
for could re-infect the network and therefore needed to be identified in
To start the process, Microsoft Information Scan was used to remove the
worm from Exchange. Next, Trend Micro Inc.'s ScanMail antivirus software
was deployed to stop the worm from breaching the perimeter. Wood said he
used Tivoli products to scan his enterprise's 100 servers and 1,750 desktops
in 128 locations for infected systems and to clean those in need with a
distributed package of scripts created by the team. Using Windows NT logging,
the scripts made sure an unknown machine could not log on to the network
until it verified its end-point location.
Following that, new Sophos antivirus software was installed on remote machines
from a centralized system. New rules were added to pay close attention to
potential worm-carrying agents such as Visual Basic Script files, ZIP files
and .exe files.
In less than a week, Wood said, his system was 90 percent to 99 percent
up and running and clean, without a single project timeline or federal appropriation
"I don't think any of us really expected this kind of virus attack to be
as smartly written — as virulent," Wood said. "I'll admit, personally, I
was caught by surprise."
Wood said security technology is playing a bigger role in his department
these days. He is determined to have checks and tools in place so he won't
have to dodge a similar bullet.
"A lot of us are getting a little bit long in the tooth and gray in the
beard," Wood said. "We need to pass something to our successors for them
to understand how we dealt with this process."
NEXT STORY: Study zeroes in on public/private partners