A lesson in virtualization
With the launch of the VTrckS initiative, the CDC solidified the use of virtualized technology within their organization
The Centers for Disease Control and Prevention is a frontline agency that protects Americans against numerous health threats. One of its main jobs is to make sure vaccines are distributed to more than 100,000 doctors and clinicians around the country.
Until recently, the system CDC had in place for that was a mess. Distribution was handled through 64 state, local and territorial health departments, each of which had its own methods for tracking the vaccines. Many of them were manual and paper-based. There was potentially great waste, and no real idea of what was in the pipeline at any given time.
With the vaccine distribution system only looking to get more extensive and complex over time, CDC officials knew things had to change. In 2003, they launched the Vaccine Tracking System (VTrckS) initiative, the National Center for Immunization and Respiratory Diseases’ first attempt at a comprehensive IT program for vaccine ordering and distribution.
VTrckS went live with six pilot programs at the end of 2010. CDC officials expect to launch the final system in April.
It was also the program that helped solidify the use of virtualization technology throughout CDC.
“The lesson we learned from VTrckS is that we can virtualize almost anything and everything,” said Steve Warren, project officer at the Mid-Tier Data Center in CDC’s Information Technology Services Office.
The problem for CDC was that VTrckS was given a very tight timeline to be up and running. Officials had a good plan and a strong project management group in place, but nevertheless, other things would take time, which is where the decision to use virtualization came in.
Initially, Warren said, officials thought that although some things could obviously be virtualized, a good chunk of VTrckS would have to run on physical servers because of the experience CDC had with other enterprise resource planning projects. (VTrckS uses an ERP from SAP.)
In the end, he said, they were able to build a big enough virtual machine, and so far, given the performance of the early VTrckS, that decision has been borne out. They knew there were risks in going with a completely virtualized environment, but in this case, they decided that the benefits outweighed those risks.
“What we were more concerned about was being able to quickly increase the number of virtual processors and virtual RAM,” Warren said. “If we had undersized the whole thing, we would still have the flexibility to increase at least those two components.”
It was understood from the start that a threshold could be reached with virtual machines that would have required a move to a physical system. However, given the advances in server CPUs that contain multiple cores, the increases in memory size and the advances in the VMware virtualization software that CDC uses, that threshold keeps getting pushed up.
Although CDC officials had been involved in virtualization before, VTrckS was considered a unique application because of its size and complexity and would not be a run-of-the-mill deployment, Warren said. And there was an urgency about it because of the tight timelines, with a need to get that first environment built so the development team would have something to work with.
Because of previous virtualization projects, Warren said officials knew they could virtualize various VTrckS components, but the real question was one of scale: Could the CDC team virtualize SQL servers that would be large enough to handle VTrckS’ needs?
The answer was yes. Furthermore, it was able to create virtual machines quickly, which was another thing that turned out to be very helpful, Warren said. The team could use existing physical capacity at CDC to build the first environment and had enough capacity on hand from virtual farms to get VTrckS going immediately. And then it was able to stay ahead of the demand by creating virtual machines.
If physical servers had been used, there was a big risk of buying equipment that was either too small or too big for the project. Virtualization allowed the CDC team to take a Goldilocks approach and build the virtual machines to a point they thought was needed and then adjust them as the test results came in.
“If a particular server started out with four virtual CPUs but really needed six, we could crank it to six in less than a day,” Warren said. “If a server had 8G of RAM and needed 12, we could also rightsize that very quickly.”
Given the size and complexity of VTrckS, the lesson learned from all this “is that we really don’t question anymore what we can and can’t virtualize,” he said. “When we consider beginning a new project, we first look to virtualizing, and if we don’t, there has to be a technical, documented reason why we can’t. And we use VTrckS as the benchmark.”