Cleaning up the clutter

To Scott Carson, the government's disorderly data storage habits are creating a wellspring of work for his employer, IMS Systems (IMSS).

In fact, 40 percent of the storage integration firm's revenues so far this year have come from storage consolidation work performed for customers such as the Federal Communications Commission and the departments of Transportation and Health and Human Services.

Reasons for consolidating storage vary by client, but Carson, IMSS' chief technology officer and managing director for technical services, sees simplified management and lower costs as the primary reasons for consolidation.

The alternative -- fielding lots of servers with direct-attached or internal storage, each managed separately -- has a high total cost of ownership that can be effectively attacked by consolidating, he said.

Aggregating distributed storage into fewer devices reduces the management burden and enables raw storage capacity to be allocated more efficiently. But consolidation is hardly a one-size-fits-all solution, and agency officials need to choose an approach carefully.

Options include deploying stand-alone network-attached storage (NAS) appliances, super-sized shared disk arrays and pooled devices connected in a storage-area network (SAN). Examples of each are plentiful throughout government.

Standard configurations

Simplifying management was the goal behind the Army National Guard's recently completed storage consolidation project, according to Lawrence Borkowski, a deputy in the guard's Information Technology Plans, Programs and Policy Division. Guard officials sought to simplify IT management by deploying "standard configuration management for storage to all our 54 states and territories," Borkowski said.

Storage supports systems that handle personnel records management, finance, logistics, contracts, e-mail and other applications.

In January 2002, officials formulated a set of requirements that were sent to 20 vendors for bids, and they received nine replies. In May of that year, officials chose EMC Corp.

The implementation of EMC Symmetrix disk arrays at guard facilities in 56 cities nationwide, plus the consolidation of the guard's Washington, D.C., headquarters' storage on a Symmetrix SAN, was carried out later that year. Using a largely homogeneous equipment solution, officials were able to capitalize on EMC management and replication software products to "streamline storage administrative tasks and shift more to warfighter focus," Borkowski said.

In the process of consolidating, the guard's EMC storage investment grew from 8 terabytes to 270 terabytes. Regardless of the capital cost, which guard officials declined to provide, Borkowski said the effort actually produced $10.5 million in cost reductions, including management cost savings and system downtime reduction, thanks to data replication among sites.

The secret to the project's success was common sense, he said. "Know what you want to do, do your homework and work with a good vendor," he said.

His advice is echoed by Samuel Chung, senior systems engineer in the Army's Program Executive Office for Enterprise Information Systems' Project Office-Civilian Personnel Regionalization.

In 1999, Chung's team deployed 10 NAS solutions for Army Civilian Personnel Operation Centers (CPOCs). Network Appliance Inc. was tapped to host a distributed human resources application and database, with the company's SnapMirror software enabling enhanced data consolidation and protection.

Last year, CPOC officials decided to set up the new centralized Army Civilian Data Center (ACDC) in Illinois, Chung said. Once again, the Army turned to his team to consolidate and centralize the data, and again, he turned to NetApp for the solution.

"The biggest challenge facing the Army civilian personnel community at that time was to move the disparate 100G databases to ACDC and merge all of the databases during a single weekend," he said. They "could not afford to waste the time of more than 3,000 personnel specialists while mission-critical databases were being merged."

The huge geographical distances between CPOCs, which are located in Korea, Alaska, Europe and seven other locations, complicated consolidation. Copying all databases to tape and hand-delivering the tapes from each CPOC to ACDC to keep the mission-critical data safe was not impossible. But as an alternative, Chung used SnapMirror to copy and send the information in all 10 databases to ACDC across Army networks.

The first step was a proof-of-concept test. "We successfully demonstrated the usefulness and viability of moving a 100G database from one CPOC to ACDC within five minutes" after creating a mirror of the data, he said. "After the successful demonstration, we were given an authorization to use the [NetApp] SnapMirror software to move all 10 databases during migration and merge."

At first, the Army civilian personnel still wanted to send tapes to ACDC as a backup. But the test results convinced them otherwise. "After my team successfully moved and merged 10 databases within a single day, even before the tapes could be shipped to ACDC, the Army civilian personnel community decided that this backup plan was no longer needed," Chung said.

Consolidated backup

Centralizing backup processes is another storage consolidation strategy that can provide immediate benefits, according to Oscar Ernst, a storage consultant and chief executive officer of SouthernLink-IT.

Ernst said a major objective in a recent storage consolidation job the company did for a commercial client involved using two large tape libraries as targets for backups that were previously performed using locally attached tape drives.

"In over 20 years' experience in the IT industry, I never saw a project with a shorter payoff time or one that was capable of providing such peace of mind to a client with respect to reliable, pain-free backups," he said.

Tape backup problems were the impetus behind the FCC's consolidation project, according to David Miller, an IT specialist for the agency in Washington, D.C. The old routine of backing up multiple servers using local tape drives installed on each was tedious.

"We needed several people to perform the backups and several backup people to cover the primary people when they were out sick or on vacation," he said. "Dealing with it was horrid."

FCC officials opted to use a SAN fabric to interconnect storage and separate it from servers so that they could use two automated tape libraries as centralized backup repositories. Now, one person performs all backups, Miller said.

According to IMSS' Carson, disaster recovery is the second most common reason given for consolidation, especially since the Sept. 11, 2001, terrorist attacks. "Consolidating storage in a SAN gives you many options for data replication and downtime reduction," he said. IMSS served as an integrator for the FCC's project, which predated the attacks by a month.

Miller said he cannot quantify the reduction in downtime that has occurred since he incrementally deployed the 12 terabyte SAN at the agency. The new network was implemented at about the same time that the FCC moved from Novell Inc. servers and networks to those from Microsoft Corp., "so it is difficult to compare downtime levels or to link improvements to the SAN," he said.

Miller constructed the SAN, with IMSS officials' help, with cost in mind. He chose smaller, 16-port Fibre Channel switches, which were interconnected with careful attention to bandwidth conservation. He used his integrator to test everything first, a move necessitated by the tendency of SAN vendors "to behave a lot like used-car salesmen," he said.

In the end, his 12 terabyte SAN was homogeneous within component types -- all storage was purchased from one vendor, all switch gear from one vendor, all host bus adapters from one vendor -- to ensure product compatibility.

A side benefit of consolidated storage is that each server can treat the storage allocated to it like direct-attached storage. File systems manage storage allocated to their use. This proved useful as servers were changed from Novell to Microsoft.

Tape libraries are the only hurdle Miller has encountered. "We have zoned our two large libraries so that each server is responsible for its own tape resources," he said. "The problem is, if one server locks up a tape drive in the library, it is sometimes a challenge to fix the problem."

Initially, he had some difficulty getting all project groups at the FCC to join the centralized SAN. "Now we have to beat them away," Miller said. "We don't have enough space for everyone."

A storage consolidation project done correctly does not have to interfere with applications or the people who rely on the storage resources.

Traditional "direct-attached storage addresses the needs of the applications, and since most SANs today are extensions of the [direct-attached storage] model transported over storage fabric, applications needs are not disrupted, and the SAN storage consolidation works," said Robert Griswold, an independent storage consultant based in Round Rock, Texas.

Although officials at networked storage vendors describe numerous benefits to consolidation, including reduced costs for server software licenses and enhanced data sharing, Carson said these are not among his clients' most commonly cited goals.

"I would have thought that data sharing would be much more important than it is, and there are some good products that facilitate data sharing out there once storage is consolidated into a SAN," he said. "But maybe it's too early for this to be a compelling driver."

For now, reduced administrative burdens and enhanced data protection continue to be the main arguments for storage consolidation.

Toigo is an independent consultant and author specializing in business automation issues. He can be reached via his Web site at www.toigoproductions.com.