Making EVM matter
Earned value management has been around for decades but is now becoming an essential part of how agencies manage projects. Why should you care?
Read AFFIRM's white paper on EVM
Until recently, earned value management — a system for managing projects — was not well-known among federal agencies outside the Defense Department. But this year, the Association for Federal Information Resources Management selected EVM as the topic for its 11th annual Emerging Issues Roundtable.
During a wide-ranging discussion, experts offered their insights on how to effectively use EVM to manage projects and systems. Christopher J. Dorobek, Federal Computer Week’s editor in chief, led the discussion.
The participants were:
Brian Burns, chief technology officer for infrastructure at the Interior Department. Previously, he served as deputy assistant secretary for information resources management (IRM) and chief information officer at the Office of the Assistant Secretary for Indian Affairs. He has also worked as deputy assistant secretary for IRM and deputy CIO at the Department of Health and Human Services.
Manny De Vera, program director and practitioner for service delivery and project management at the General Services Administration’s Federal Acquisition Service. For the past 10 years, he’s also been a part-time instructor for the Agriculture Department’s graduate school on project management and EVM.
J. Davidson Frame, academic dean of the University of Management and Technology. For 20 years, he was in the Management Science Department at George Washington University, where he established a project management program and was director of the Science Technology Innovation Program.
Odelia Funke, director of the Mission Investment Solutions Division at the Environmental Protection Agency.
Sally Good-Burton, chief of Interior’s Information Technology and Portfolio Management Division, which handles capital planning and EVM. Previously, she was director of IT investment management at the Department of Housing and Urban Development.
Dave Muzio, procurement policy analyst in the Office of Management and Budget’s Office of Federal Procurement Policy. He serves as the agency’s focal point for EVM and has been involved with EVM since 1967.
John Rehberger, program manager for enterprise architecture at the USDA. Five years before joining that agency, he was a capital planning theoretician at the Government Accountability Office.
Karen Richey, senior IT specialist at GAO’s Center for Technology and Engineering. She started using EVM while working on a major weapons systems program at the Naval Air Systems Command. She then went to the Naval Center for Cost Analysis, which handles independent cost estimating. At GAO, she is working to further incorporate EVM into the agency’s audit findings.
Robert Rovinsky, program director for the Federal Aviation Administration’s Strategy and Investment Analysis Division. He reports to the FAA’s CIO and is responsible for Exhibit 300s. He is co-leader of the FAA’s Earned Value Management Initiative. Before joining the FAA, he was responsible for the Fairfax County. Va., Office of Research and Statistics. Previously, he spent 11 years at the USDA.
Dennis White, practice manager for performance management at Robbins-Gioia, a project management consulting company. He has held many consulting positions and has also worked for Rockwell International and served in the U.S. Air Force.
Sally, when you talk to people who may not know about EVM or fully understand it, what do you tell them?
GOOD-BURTON: I tell them it’s just a better way of managing their resources.
For years the concentration was just on how agencies obligate their money, just tracking the spending. That’s all the managers really wanted to know — did they spend their money by the end of the fiscal year? But they didn’t tell you what you got for it.
So EVM tells you — by planning and setting the milestones — whether you got the value for what you’re paying for. I think people are starting to understand that.
ROVINSKY: At the FAA, we originally wrote a policy statement that clearly stated that you had to use it, but when we wrote our guidance, the original draft that was put together said, “Well, you should consider using EVM. It’s a good tool but, by the way, it’s a lot of work. It’s a lot of trouble. It takes a long time and will add more money, but consider it anyway.” I said, “No, that’s not the way we’re going to write our guidance. It must be used depending on the size of the project. It’s a requirement.”
John, what’s your perspective at the Agriculture Department? Do people understand what EVM is and would they be using it if they weren’t being forced to use it?
REHBERGER: I think it depends on who you perceive the beneficiaries of EVM to be. We teach EVM within the construct of overall good project management. And it really is just one element of good project management on par with scope management, communications and other such factors. We have a few top-notch managers who understand EVM and take it upon themselves to do it. Others, you’ve got to twist their arms. I’m not going to lie to you. The problem with twisting their arms is you get a set of perverse incentives. You get people playing games with EVM when the goal is just to report the variances as close as you can to OMB and then just check that box off.
David, what do you think as we sort out whether there should be a mandate or not and whether EVM works well for all agencies or not? How can it be presented to project managers?
FRAME: When this was first rolled out and really being pushed on the contractors, there was a lot of resistance. But we have to remember the historical context. There was something called DOD Instruction 7000.2. Just the name itself kind of freaks people out. And when General Electric and similar organizations had to implement it, they had to revamp their total accounting system. The one thing that you have to remember if you’re involved with earned value is that the toughest thing to measure is actual cost. Earned value is a piece of cake.
It’s basically a lot of good-sense management, and I think it’s much easier to implement now.
A year and a half ago, I was part of a team that investigated EVM implementation at the Energy Department. We were an independent research team and were told to go in there and savage them if necessary. They had a requirement: EVM was mandated for projects worth $20 million or more. For projects less than $20 million, EVM was not mandated. On every one of the smaller projects at DOE, everyone opted to employ earned value because it wasn’t that difficult to implement and they found there was tremendous value.
Odelia, why don’t you talk about EVM use at the Environmental Protection Agency a little bit?
FUNKE: We’ve adopted EVM pretty broadly for our business case process. I see in our distributed system that we have responsibility and budgets controlled in a distributed fashion agencywide. I think it could be a good tool for senior management oversight.
One of the issues I hear is that scalability has not been solved yet. So on the one hand it affects small contractors because this still seems pretty formidable and expensive to them.
I think that it takes training senior managers to understand what they can get from it and the kinds of questions they need to ask depending upon the deviations they’re seeing, and how
it might not be a bad sign if people have, in fact, adjusted their plans because of changing
Odelia mentioned the issue of scalability. Has anyone else experienced that?
REHBERGER: I jokingly tell people that EVM is one of those unique areas in IT where they are more concerned about scaling down than they are about scaling up because it has been proven it can scale when the numbers are big and the importance is high. What’s the lower threshold on that? I’m not sure it’s worth it for a million-dollar project. A $5 million project, I’m not sure it’s worth it there. The supporting evidence I have is that every organization has basically a different set of criteria for when ANSI 748 compliance is required. If everybody would gravitate toward a certain set, then that would be the consensus as to what the criteria are.
So, to me, scalability is an issue. It’s very much an issue that is being sorted on a daily basis. I think OMB was wise to present the issue in its memo last year to sort of decentralize it. The agencies have to create their own criteria. However, the feedback we’re getting recently is, for example, that everything has got to be reported — Exhibit 300s have to use EVM even though that violates our own criteria. So my concern is, where is the downward scalability?
DE VERA: Earned value doesn’t cost anything. It saves us money. It may cost you if you don’t have a good management system. We consistently have contractors who can’t manage because they don’t have a system good enough to know where $20 million is going. So the key is we need to make sure that if we’re going to spend that money that we have a system that tracks how well they’re spending the money.
All project managers should at least start with the notion to use earned value to the necessary degree and apply security. If you need a lot of security on a project, you use it. You don’t need a lot of security on small projects, and it’s the same concept when you apply earned value. It’s just good judgment and project management to use the degree of earned value tools that would apply.
ROVINSKY: Our chief procurement officer, who was concerned about the cost of EVM, approached us. Her experience, as shared by many people in the acquisition shop, is that people ask a lot of money to do it. It’s an add-on cost. They didn’t see the value. We went back and did a study. The FAA had evaluated all of its major programs as to how well they were using EVM and rated them red, yellow or green.
So we evaluated our programs with those three categories and then looked at the percentage that they were spending on program management as a percent of total project cost. We found that those programs that we had rated red — not using much EVM — cost more than those that
we had rated yellow. And those that we had rated yellow cost more than those that we rated green.
So we went back and said, “Using EVM is not adding cost. It’s actually reducing the overhead of managing a project.” Of course, her response was, “That’s obvious.” But it wasn’t obvious until we did the work. I think that’s the kind of data that is helpful to be able to go back and show that, indeed, this is not an add-on cost. It actually saves money.
MUZIO: It seems to me that one of the clear benefits of EVM is the idea of articulating performance goals. I’ve been at the agency a long time, and one of the abiding difficulties is people start a project without clearly articulating for themselves what the performance goals are. And, of course, what you don’t understand, you cannot possibly measure.
So, scalability is a task well worth undertaking, and I certainly would agree with the comments that, as an approach to managing a project, this really articulates some fundamental concepts that are valuable for any projects. Teaching scalability and how to use EVM are equally important challenges.
I keep coming back to this question: If I’m a fed and I’m running a project, why wouldn’t I use EVM? This gives me some kind of safety net. It’s a process that directs where I should go, and it helps me identify if I’m having problems. Yet we all know that it’s not the way most projects are managed.
WHITE: That’s when I usually ask, “Why wouldn’t you do this?”
And what do they say?
WHITE: It’s scary. Ask somebody to take a look at ANSI 748, which is the standard for EVM. We don’t have 35 criteria anymore. We only have 32, but they are kinder and gentler. There has been some maturity in that. We’ve actually seen those 35 criteria accompanied by some 130 to 140 check-mark questions.
There are only 32 criteria now, and those are not nearly as directive and prescriptive as they used to be. They are much more broad in context and much easier to accommodate.
DE VERA: We need to demystify this not only among the project managers but among the executives and stakeholders who touch some of these programs because I think they don’t understand it.
Executives and stakeholders want the program manager to get the training but I’m not really sure they know how to use it or are aware that this is a good thing to help them at the executive level because we start to use our EVM and technocrat jargon.
FRAME: Part of this is moving beyond the 32 criteria and getting people to go back and recognize that this is basic common-sense management.
WHITE: Typically, we have a tendency to manage based on some dollar figure, our budgets, or we manage our schedules to when something is going to be done. I think one of the biggest benefits in earned value is it forces us to focus on the scope of the work.
What we are really dealing with is the content of that work, the cost information and the scheduling information. That is where your prime benefit comes in — when you are able to focus on what’s getting done or not getting done rather than where you’re at in a budget scale or the schedule lags.
Both are important, but if the scope of work doesn’t get done, the project is down the tubes anyway.
DE VERA: I sincerely believe that the three communities — the financial, the acquisition and the IT or project management teams — are getting on board with EVM, and I think that the last piece of this are the executives.
I think they need to know a little bit more about earned value so that they can help with the analysis. If we bring them on board, I think it will help us at our level.
We always talk about projects that have failed, and I think there are probably not many systems that we can just decide we’re going to close down and move on. Can anyone talk about instances where EVM has either helped them save a project or decide to end it?
BURNS: As an executive overseeing a lot of these things, it has come down to the red and green view of the charts. It has forced me to choose to continue, fix or kill a project.
In one situation, we had a project that was behind schedule and also over budget, but that project was fixable in the sense that we quickly identified what the issues were and took appropriate action. In this particular case, it also had some contractual issues that we had to deal with, and we were able to go back to procurement and work some of those things out.
RICHEY: At GAO, we love to use EVM data and be able to use it early on to see if we should continue or end a project. Then we can present it to our management at GAO, and we can give it to the decision-makers at Congress to let them know ahead of time.
Everybody is under extreme budget pressure. Everybody needs to get their programs funded, and we see EVM as a great tool. It benefits us because it provides accountability. It very much exposes the problems. You can get down to the problems and the root to quickly analyze. The best you can do is give that information to decision-makers and let them decide whether or not to continue the program, but at least you can do your job of giving them objective data to make that decision.
FRAME: EVM gives you very early warning — very early warning.
I want to talk momentarily about the shadow side of earned value. It is not that tough to
At one of the agencies I worked with, we had these astonishing success stories of the two or the three lines — the earned value line, the actual cost line and the plan cost line. All that being right on top of each other, and you are saying, “This is a miracle.” Then you talk to people on the team and they’re saying, “This is a nightmare.”’
MUZIO: We understand you can game it. You can game any kind of a system. In earned value, though, there’s always a catch point — a day of reckoning.
ROVINSKY: There’s no value in continuing to game the system or lying. GAO is developing a lot of expertise to come in and review things. OMB has expertise. There is a lot of expertise out there that’s going to reveal if you are gaming the system. So it’s not going to work just as trying to game the Exhibit 300s did not work a few years ago.
MUZIO: The thing is for the government to understand that the customer is not the agency. The customer is the taxpayer. When they go into these projects, they’re basically going through the system, and they’ve promised management that I can deliver this program, which meets the strategic goal and objective need in this time frame, which will save us money because we can shut down for this amount of money. So they’ve made a promise, and yet most of the promises when they go into them aren’t real promises. They’re just getting into it to get the money, and I think we have to change the mind-set to have agencies understand and put the discipline in to develop the goals that are risk-adjusted and are realistic so that management can say yes or no, and they can actually deliver what they promised to do to make this agency a better server to the public, and that’s not a mind-set that they have. It’s overrun, send some more money, and we need to change that mind-set.
RICHEY: We are embracing EVM. We actually are putting out federal guidance. It’s a cost assessment guide that we’re working on right now. It’s going to take earned cost estimating and best practices, leading practices in cost estimating, and what puts together a credible cost estimate, and we’ll link it to EVM to say, “You do an estimate, it has got risk in it, it gets awarded into a contract, and you monitor the progress of it in a contract using EVM.” And we want to be able to alert the auditor for what to look for in an EVM system to see if it is valid, if it is getting good data. If it passes that test, then what is done with the data? How to take it to do independent assessments, how to do the drill downs for risk analysis, looking through the Format 5s, taking what kind of variances you’re finding in the EVM system and linking them back to the risks that were identified in the original cost estimate.
So this guide is something we’re in the process of developing. We are opening it to anyone who has an interest in helping us develop it. We have a community of experts right now, mostly DOD officials, so we’re trying to involve more civilian agencies. We’re trying to get it out in the fall.
FUNKE: My last words: I have just a greater optimism that EVM can be used flexibly and should be used to manage not only the project scope but also the change in scope, which I think is important because it’s not just a problem in the government. It’s definitely a problem in the private sector as well, the scope creep that sometimes can overtake projects.
Second, we probably need to pay more attention to educating executives about what this tool is, what it can do and how to use it to our best advantage.