In pursuit of consolidation

Michigan’s data centers were vulnerable, but not all managers welcomed consolidation

Related Links

Deft consolidation

Michigan is proud of its three state-of-the-art data centers, which now handle most of the data processing for the state’s 19 agencies. One center is for data processing. Another center handles emergency services such as enhanced 911 communications. The third one is a disaster recovery center, which the other two centers could use if a disaster disrupted their operations. If you’re a state official and would like to visit, chances are the Michigan Department of Information Technology would be happy to accommodate you.

But if you had asked for a tour of the state’s data processing facilities as recently as five years ago, you might not have received as hearty a welcome as you would today. A complete tour would have required a bit of traveling.

“We had 19 different centers, each with varying degrees of efficiency,” said Ken Theis, chief deputy director of the IT department. Although some of the data centers were clean and well-equipped with modern ventilation, dual-power supplies and state-of-the-art servers, others were unprotected. Some had backup batteries that had been burned out and power cords that were plugged directly into wall sockets. Others relied on ceiling fans for ventilation. Many were cluttered with strands of cables that were difficult to track. One center had electric cables running along the outside of the building. Anyone with a cable cutter could have shut down the entire data center operation for hours or longer.

In addition, many of the centers used unique tools, which necessitated specific training at those sites. And in many instances, servers and storage were underused, adding unnecessary costs to the state’s IT budget.

If that weren’t enough to get data center managers to cry for help, budget problems in 2001 forced the state to cut one-third of its data center employees. “The old system was too resource- and labor-intensive,” said Patrick Hale, the IT department’s director of infrastructure services.

Michigan’s answer to the problem was to close 17 data centers. Although merging services isn’t a panacea, it can provide efficiencies of scale; better use of resources; and improved management, customer support and uptime. But the challenges to achieving those benefits aren’t trivial. For example, most states that are considering a consolidation project will have to work hard to achieve buy-in from clients, executives and IT employees. Even though many of Michigan’s data center managers were in distress, they weren’t overjoyed at the prospect of transferring their data center activities to a central megacenter. “There are always fiefdoms,” Hale said.

Make the case for consolidation 
Hale and Theis said IT departments must earn some street credibility if they hope to garner the support they need to consolidate data centers. “You have to prove yourself, provide evidence that the new center will be better than the previous ones and that there won’t be any problems with the move,” Theis said.

Michigan’s IT department leaders reasoned that if a picture is worth a thousand words, an actual visit combined with pictures could be many times more valuable.

So the department bused clients and state officials to the new data center facilities and gave them VIP tours. Then they showed them photos of their current centers — with the poor ventilation, power hazards and disorganization. Theis and Hale handed out cards with their home telephone numbers and invited people to call them with questions at any time. “It was a watershed event,” Hale said. “Most people had no idea how vulnerable their data centers and their data were.”

And realizing that no one can resist a bargain, the IT department made a commitment to charge agencies no more for their data processing than they were paying and promising an eventual, if not immediate, reduction in data center costs.

Nevertheless, data center managers still had concerns about the consolidation plan. Theis said he knew that a misstep could thwart the whole project. He also knew that if one data center went through a successful conversion, word of that success would spread. 

Extend the storage-area network
To say that Michigan’s IT department took the necessary time to do the conversions right is a bit of an understatement. Officials began the consolidation of IT resources during the administration of former Gov. John Engler, a Republican, in the early 1990s. The data center consolidation project began under the administration of Democratic Gov. Jennifer Granholm. If someone had to make a trade-off between doing it right and meeting a deadline, doing it right won, Theis said. “We were not afraid to extend schedules, even going 60 days beyond expected switchover, if we had any questions about getting it right.”

To move data to the consolidated center, the IT department extended the central data center’s storage-area network fabric to the data centers it was replacing.

 Then it replicated the data onto the SAN while the old center continued running. Only after the data was successfully transferred and tested did officials shift data processing operations to the new center.

Theis said the state completed 17 conversions without a single hour of downtime. The IT department announced in February the closing of three more data centers.

Since consolidating the work of 17 data centers, the IT department has standardized service levels for all state users. “We can view every node on the network and get alerts when anything needs attention,” Theis said.

Calculate intangible ROI
Michigan’s IT officials have not fully calculated the return on their investment in consolidation. The only hard numbers they have so far are based on the results of consolidating messaging services. Officials said the project saved the state more than $600,000 this year. They estimate savings of more than $11 million in four years. The messaging consolidation brought increased service levels and 50 percent improvement in performance.

But Michigan officials said many benefits of the consolidation, such as greater efficiency, are difficult to measure. For example, many of the state’s data centers followed the conventional practice of having servers running a single application. In one instance, the IT department replaced 40 servers with one high-end Sun Microsystems server.

Another benefit is more efficient storage. Michigan installed a SAN that spans all three new data centers. The IT department charges agencies a per-gigabyte fee for storage. Agencies’ storage costs have been dropping because of greater storage efficiency, although IT department officials said they do not yet know at what level those fees will stabilize. 

In Michigan, savings from consolidation are already manifest in messaging services, and benefits have shown themselves in virtually all systems. “Our consolidation [is] a win for everyone,” Theis said. “Costs are down, service is up, and we have a much safer center.”

Stevens is a freelance journalist who has written about information technology since 1982.
Four steps to messaging consolidationThe Michigan Department of Information Technology carried out a messaging consolidation project in four stages. That  approach  helped reduce resistance and gave employees a chance to get used to the change.

Step by step, the IT department:
1. Brought together everyone affected by the consolidation.
  • IT department officials created a technology steering committee composed of representatives from each agency and branch of state government, and they presented to the committee the business case for consolidation. 
2. Created a common e-mail identity.
  • The IT department developed a unified gateway and the common domain name, created a filter to protect the state from spam and virus attacks, and enabled the state’s primary e-mail platforms — Microsoft Exchange and Novell GroupWise — to coexist. 
3. Improved e-mail user support.
  • Officials consolidated all e-mail users under two applications: Microsoft Exchange and Novell GroupWise. The e-mail servers remained geographically dispersed, but employees who provided user support knew the systems they were dealing with. As a result, the central IT department was in a better position to respond to trouble calls.
4. Moved the e-mail servers to a central data center.
  • The central IT department moved all e-mail servers to a single hosting center.
— Larry Stevens

The Fed 100

Read the profiles of all this year's winners.


  • Then-presidential candidate Donald Trump at a 2016 campaign event. Image: Shutterstock

    'Buy American' order puts procurement in the spotlight

    Some IT contractors are worried that the "buy American" executive order from President Trump could squeeze key innovators out of the market.

  • OMB chief Mick Mulvaney, shown here in as a member of Congress in 2013. (Photo credit Gage Skidmore/Flickr)

    White House taps old policies for new government makeover

    New guidance from OMB advises agencies to use shared services, GWACs and federal schedules for acquisition, and to leverage IT wherever possible in restructuring plans.

  • Shutterstock image (by Everett Historical): aerial of the Pentagon.

    What DOD's next CIO will have to deal with

    It could be months before the Defense Department has a new CIO, and he or she will face a host of organizational and operational challenges from Day One

  • USAF Gen. John Hyten

    General: Cyber Command needs new platform before NSA split

    U.S. Cyber Command should be elevated to a full combatant command as soon as possible, the head of Strategic Command told Congress, but it cannot be separated from the NSA until it has its own cyber platform.

  • Image from Shutterstock.

    DLA goes virtual

    The Defense Logistics Agency is in the midst of an ambitious campaign to eliminate its IT infrastructure and transition to using exclusively shared, hosted and virtual services.

  • Fed 100 logo

    The 2017 Federal 100

    The women and men who make up this year's Fed 100 are proof positive of what one person can make possibile in federal IT. Read on to learn more about each and every winner's accomplishments.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group