EVM: A dash of cold water

When officials at the Social Security Administration began to integrate earned value management into their project-management drill, they did something that few agencies have done. They established an EVM program management office that’s become a major hub of activity.  As part of the office of the deputy commissioner for systems, the EVM PMO doesn’t operate in a vacuum. It’s structured to push the data that EVM generates on information technology projects up through SSA’s management anks and throughout the agency. EVM reports are published on the agency’s intranet where employees at SSA can review them. The EVM PMO also routinely routes data to the agency’s Information Technology Advisory Board — its IT executive steering committee — and the SSA deputy commissioner’s office. And every quarter, senior-level executives review EVM performance for all the agency’s programs.“EVM data and the results of the reports are available all over the place,” said Otto Immink, senior IT specialist in the EVM PMO. “It is exposed across the agency and utilized by all levels of management.”SSA is one of the few agencies pursuing a program-level approach to EVM, said Karen Richey, senior cost analyst at the Government Accountability Office. “They’re not just doing it at the contract, but they’re trying to include all [management levels] in house and roll it up into a full-out program level,” she said. “Even DOD isn’t doing program-level EVM yet.” However, the Federal Aviation Administration has taken a similar program-level approach to EVM, she added.Getting EVM out of isolation in the project manager’s shop and into the executive domain is one of the fundamental challenges of deploying EVM in government, said David McClure, a research director at Gartner’s Government Group. SSA officials have done an exceptional job of achieving that, said McClure, former director of IT management issues at GAO. “It’s a good example of an organization that has taken EVM data and put it into the IT governance process so it’s not buried down at the project-management level,” he said. “It’s actually raised up to the level of discussion where it should appropriately be.”Moreover, SSA officials have added clarity to the EVM data so executives can understand the reports. “The formulas, calculations and indices are going to be Greek to somebody who doesn’t know what all this means,” McClure said.In its reports, the PMO explains what the indices mean with a simple system that uses red, green and yellow lines. “We measure the programs based on the standard EVM variances, such as cost and schedule variance and variance at completion, but we do it in nontechnical terms,” Immink said. For example, “if you have something that is plus or minus 10 percent off from where your established baseline was, you go yellow — that’s a caution. If it’s red, it better not be the first time management sees it. Red is an indicator that more serious attention needs to be placed on the program.”Presenting performance information to senior managers in this way increases executive accountability, McClure said. “If it’s that transparent, it’s going to be difficult for someone to say, ‘I never knew that this project was in trouble.’ ” In September 2006, SSA earned a good review from its inspector general for EVM. The IG’s audit found that SSA had implemented a system to manage major IT projects in accordance with guidance from the Office of Management and Budget. OMB mandated two years ago that all agencies use EVM for major new IT projects, reflecting the Bush administration’s emphasis on program performance and outcomes in the President’s Management Agenda.EVM is a performance management technique that measures a project’s progress against cost, schedule and technical, or performance, baselines. The EVM term for progress is “earned value.” Dennis White, Robbins-Gioia’s EVM practice area manager, said EVM centers on meeting project objectives, making cost only one element of the measurement metric. That approach contrasts with traditional project development, in which cost is the primary measure. “Once [an EVM] system — and system is defined as people, processes and tools — has been established and matured, the biggest benefit is going to be enhanced planning,” White said. “You will know what it takes to do things. You will get better at estimating and can do more realistic scheduling. You will be able to set up performance from the front end, as opposed to measuring everything from the back end.”Experts agree that for many agencies, implementing EVM has been a struggle, and the reasons for those situations vary. For some, it’s fearing the unknown, Richey said. “In civil agencies, a lot of times people are afraid of EVM,” she said. “There’s a lot of fear that it’s going to be a lot work, and there’s probably a misconception about the amount of overhead it creates for a program.”Richey added that many officials don’t understand that EVM is simply good project management practice. She said most of the 32 guidelines in the National Defense Industrial Association Program Management Systems Committee’s EVM System Intent Guide, a primary reference work for agencies, are steps agencies should take to manage any project, no matter how large or complex. “You’ve got to define what you are going to do, you’ve got to measure it and look at what’s causing any variances,” she said.Jennifer Echard, a senior cost analyst at GAO, said the absence of an oversight agency could be an impediment to civilian agencies’ EVM efforts. The Defense Department has the Defense Contract Management Agency, “which can go in and certify their EVM systems and do analysis,” she said. “There’s a lack of that on the civilian side.”A deficit of EVM training also is hindering agencies, some observers say. “It’s been a long and uphill road to get people adequately trained and experienced” in using EVM, McClure said. “There still is not enough training of EVM analysts and managers, and that continues to be an issue.”White concurred. “The lack of experienced and knowledgeable [employees] results in policies that aren’t enforceable and strong enough to make the whole set of management controls work,” he said.McClure said he is optimistic EVM will be demystified once agency officials gain experience in using EVM and understand how it works. “It’s not mystical. It’s not magical,” he said. “It’s been around for quite some time. It’s straightforward in terms of the calculations and the tracking of the project. It’s elevating the transparency of what’s actually happening with your resource expenditures and your scheduling and asking you if you need to make some changes to be more in sync with what you original expectations were.”Indeed, the potential benefits of using EVM are abundant and should outweigh what some agency officials see as the onerous task of having to meet an OMB compliance mandate, experts say. At SSA, for example, EVM provides the agency a reliable set of data that it uses to make management decisions and oversee IT programs, Immink said. “And it does so in a unique way. EVM is the only monitoring tool that I’m aware of that integrates your baselines schedule and performance against that schedule and costs incurred in the performance against the schedule.”  Overall, EVM has helped improve program management at SSA. “We’ve always had what we consider good program management practices here,” Immink said. “EVM has added another layer of rigor and discipline.”That is not to say EVM is a panacea. Experts agree that EVM has limitations.“It’s great [for] the transparency it gives in terms of cost and schedule and performance of projects, but [it] still doesn’t tell you whether the outcome is the right one from a business perspective,” McClure said. “I’ve seen lots of examples where projects are rebaselined from their original cost, schedule and performance targets, but there’s no explanatory factor given as to why the project is being rebased. So you’re just negating the value of EVM. You should be using it to educate your organization on why those variances are occurring.” Accurate data is also crucial for successful EVM efforts. “One haunting factor for earned value management is the quality of the baseline data for the project,” McClure said. “If you don’t have good, initial baseline data on cost and schedule, then this tool becomes more difficult to get value from. Without good data, you’ve got a great tool, but you’re not necessarily going to have an accurate picture of what’s going on.”Even accurate results, however, don’t guarantee EVM success. “A successful EVM implementation is measured by how heavily the resulting data is relied on,” Immink said. “You generate these fancy reports, but if it isn’t relied on for use in management and oversight decision-making, then you aren’t there yet.”