Making EVM matter: The complete transcript
Federal Computer Week and the Association for Federal Information Resources Management recently convened a roundtable of government and industry officials to discuss earned value management and its potential in government. Here is the full transcript of that discussion:
Association for Federal Information Resources Management
Emerging issues forum 11th annual roundtable
Earned Value Management
Brian Burns: Interior Department’s Chief Technology Officer for Infrastructure. Prior to that, he served as the Deputy Assistant Secretary at IRM and Chief Information Officer at Interior’s Bureau of Indian Affairs and, prior to that, the Deputy at IRM and Deputy CIO with Department of Health and Human Services.
Karen Richey: General Accounting Office. She started off at Naval Air Systems Command doing earned value management for a major weapons systems program. She then went to the Navy Center for Cost Analysis, which does independent cost estimating. At GAO, she has been working to incorporate EVM more into the organization’s audit findings.
Sally Good-Burton: Chief of the portfolio management division in the Interior Department. Her division includes capital planning and EVM. Prior to that, she was the director of IT investment management at the Department of Housing and Urban Development.
Robert Rovinsky: Program director for the Federal Aviation Administration’s Strategy and Investment Analysis Division. In that post, he reports directly to the FAA’s chief information officer. He is responsible for the Exhibit 300s and he co-lead the Earned Value Management Initiative at the FAA. Prior to coming to the FAA, I was responsible for the Office of Research and Statistics at Fairfax County. And, before that, I spent 11 years at the Agriculture Department.
Dave Muzio: Procurement policy analyst in the Office of Management and Budget’s office of e-government. He serves as the focal point in OMB for EVM and has been associated on and off with earned value since 1967.
Dennis White: Practice manager for performance management at Robbins-Gioia, a project management consulting company. Prior to that, he had other consulting jobs . He also worked for Rockwell International and was in the United States Air Force.
Odelia Funke: Division director at EPA.
John Rehberger: Program manager for enterprise architecture at the Agriculture Department. Five years before coming to USDA, he was a capital planning theoretician at GAO.
David Frame: Academic Dean of the University of Management and Technology. For 20 years, he was with the Management Science Department at George Washington University, where I set up their project management program and was director of the Science Technology Innovation Program.
Manny De Vera: Program director and practitioner for service delivery and project management for the General Services Administration’s Federal Acquisition Service. For the last ten years, he’s also been a part-time instructor for the USDA graduate school on project management and earned value.
DOROBEK: Earned value management has been sort of a buzz word over the last couple of years. I was a little shocked to hear that somebody has been working on it since the 1960's. David, why don't you talk a little bit about what is earned value management, why is this important and why is it particularly in vogue now?
MUZIO: Earned value management is basically what we have found looking across the world is probably the best project management system that we could identify. It has been around a long time. It hasn't been mandated outside of the Defense Department until the last few years. We need it because we're on kind of what I call a continuing journey to improve project management in the federal government.
Back in 1969 the Commission on Government Procurement identified major acquisitions and major systems acquisitions as a major problem. The agency is not able to manage the programs close to cost schedule and performance goals. In 1974 then the OMB Circular A109 was written, which put some of the disciplines in place, and Defense picked that up but the civilian agencies did not.
In 1994, the Federal Acquisition Streamlining Act required that agency heads establish or review cost schedule and performance goals for major acquisitions and meet at least 90 percent of those goals. FASA also required that any time a program is not within 90 percent that, the head of the agency needed to do a review to determine whether they ought to continue or stop. That statute gave Office of Federal Procurement Policy and the Office of Management and Budget requirements to look at the policies and to develop new ones. Of course, 1996 the new plan came along, which gave the director of OMB direct responsibility for setting policy and that's where we have come. So we're on this journey.
Initially we said to agencies and policy performance based management earned value or similar. So over a number of years we looked very carefully at what people were using as for similar and determined that there really wasn't anything that provided the visibility to managers at the program level, the agency level, OMB and even congress level as to where a program was, and that's key to what we're looking for. Where is your program and can you correct--take corrective actions to bring it back to the goals or at least bring it back close?
DOROBEK: Sally, as you talk to people who may not know--they hear about EVM, but does everyone fully understand what this is and, if they don't, what do you tell them?
GOOD-BURTON: I tell them it's just a better way of managing their resources.
For years the concentration was on just how agencies obligate its money, essentially just tracking the spending. That's all the managers really wanted to know -- did they spend their money by the end of the fiscal year. But they didn't tell you what you got for it.
So earned value management tells you -- by doing the planning and setting the milestones -- it lets you know did you get the value for what you're paying for. I think people are starting to understand that.
DOROBEK: Dennis, we hear much more focus on project management both from OMB. It seems like this is starting to take hold. Is it your sense that this is now something that's more important than just DOD?
WHITE: It's moving in that direction. We're getting a lot of noise in terms of interest. A lot of people are asking a lot of questions. What it is? Why it's better? What makes it different? Why should I do this? But there is still the element of resistance. They want to know where the actual requirements come from? What makes us do this? So I think it's a mix. There is a lot of interest out there but there's also still the resistance and still the reliance on requirements such as OMB Circular A-11.
DOROBEK: Is it good to mandate agencies use EVM?
WHITE: I think it is and I would like to tell you why. In the Defense Department, it was required on major contracts. And those companies were in kind of the same position as the civil agencies are today -- resistant, confused, really wanting to know why the requirements are there. Over the years they became more familiar with it and more comfortable with it.
By the time the 1980s rolled around and into the '90s, companies who had been forced to use earned value management were now implementing it across the board on commercial projects and others because it had finally gelled that it was a best practice, that it does work. So if we hadn't forced it in the first place, it might not have caught on when it did and, if we don't force it today, it might not catch on in the future.
DOROBEK: Karen, what's your sense about it having dealt with this at GAO? What's your sense about it?
RICHEY: Well, it has been received very well. Our management at GAO is very excited that we're using the EVM. Our clients on the Hill love it because we can now actually go out and give an independent assessment of what we think things are going to cost and we can also use objective factual data to refute what the contractor is saying or the program manager.
So if they're saying we're going to come in at--we're going to meet our mandate of schedule--getting something deployed by a certain date and we're going to do it within the cost, we can show trends in metrics saying there is just no way they're going to make this. Here's what historically they've been able to do and they'd have to improve 200 percent. And we go back and use data from the studies of over 700 programs showing that once you're 15 percent complete, you're not going to do any better. If anything, you'll maintain or you'll do worse. So that has been received very well. And it also, I think, has brought some more analysis to our audits. We've been able to go ahead and actually go out and say what we really think and use agency data to say that.
DOROBEK: Brian, what's your sense as to is this something that should be mandated and does it sort of force people to understand? Right now they can sort of say, 'Well, I don't understand it. I'm scare of it.' And if they're forced to use it then all of a sudden that's a major motivating factor to getting more comfortable with it.
BURNS: Well, I think in some ways it already has been mandated through the (?) Act and other initiatives like that. Another aspect of how it's maybe not mandated but it's a good idea is even with the PMA, President's Management Agenda, if you think about it. I came from the private sector and in the private sector in product development you live and die by your schedules. If you can't meet time to market, you're going to miss your opportunity. In the federal environment, we don't necessarily have time to market but we do have time to implementation of regulations and also score cards that are now in place. So our measurement is really are we going to meet in time for the PMA or the new mandates that we have. This is one mechanism we can use to actually guide us in the way of saying, "Yes, we are on target and meeting our objectives." So I think from that perspective it's a good practice if not mandated.
It also helps us if we're moving towards CMM levels and really trying to really instantiate good business practices within the organization. This is one of them that forms a baseline as a measurement of how we move forward.
DOROBEK: Bob, if this wasn't required by OMB to get towards the green, which everyone is--the nirvana that everyone wants to--would this be used as much?
ROVINSKY: I also tend to think that it would not be used as much. We originally wrote a policy statement that was very clear that you had to use it, but when we wrote our guidance, the original draft that was put together said, "Well, you should consider using EVM. It's a good tool but, by the way, it's a lot of work. It's a lot of trouble. It takes a long time and will add more money but consider it anyway." I said, "No, that's not the way we're going to write our guidance. It must be used depending on the size of the project. It's a requirement."
We do have early adopters. Those early adopters live or die by EVM. They swear by it. They've come from the private sector. They've come from DOD. But we have a lumpy organization relative to EVM and we have people for whom this is not something they'll do unless they have to do it.
DOROBEK: A lumpy organization?
ROVINSKY: Meaning that it's lumpy in terms of its--the kinds of people who will use it versus people who don't even know about it. So there are people who live or die by it and people who don't even know about it.
DOROBEK: John, what's your sense at Agriculture? Do people get what EVM is and would they be using it if it weren't being forced to use it?
REHBERGER: I think it depends on who you perceive the beneficiaries of EVM to be. We teach EVM within the construct of overall good project management. And, as such, it really is just one element of good project management on par with scope management, communications and things like that. We have a few top notch managers who get EVM and take it upon their own volition to do it. Others, you've got to twist their arms. I'm not going to lie to you. The problem with twisting their arms is you get a set of perverse incentives. You get game playing with EVM where the goal is just to report the variances as close you can to OMB and then just check that box off. So it really depends upon who you think the beneficiaries are.
If you wanted the beneficiaries to be greater OMB sort of mechanical oversight then mandate is the way to go. If you want the beneficiaries to be an enlightened set of project managers who understand the value of earned value in the context of what they're doing, then education prompting coaching is a better mechanism.
DOROBEK: David, what's your sense as we sort of sort out whether there should be a mandate or not, whether EVM works well for all agencies or not, how to get it really driven down into project managers? What's your perspective?
FRAME: Well, I think Dennis said something very important earlier and that is when this was first rolled out and really becoming pushed on the contractors there was a lot of resistance. But we have to remember the historical context. There was something called DOD Instruction 7000.2. Just the name itself kind of freaks people out. And when GE and organizations like that had to implement it, they had to revamp their total accounting system. The one thing that you have to remember if you're involved with earned value is that the toughest thing to measure is actual cost. Earned value is a piece of cake.
So I think one of the things you have to recognize is that it developed this horrific reputation as being a nightmare program that cost millions of dollars to implement. For those of you that remember, DOD Instruction 7000.2 had 35 separate criteria that you had to meet. Okay. On the one hand, the Defense Department says, "We don't want to tell you how to do your job.” But, on the other hand, we do want you to check these boxes."
But even those organizations went through the pain. They spent millions of dollars revamping their accounting systems and once they had it in place--especially in the aircraft industry in a whole bunch of areas. They applied it everywhere. They loved it.
I think the real nice thing now is that in the past 10 years people have recognized that you can take a common sense approach to measuring earned value and, in fact, people have been doing earned value analysis without knowing it for a long, long time. If you go to the construction industry--I taught earned value to construction and they were laughing at me in the class. They said, "Well, basically when you have--when you're doing a construction project and you're going to get progress payments, the progress payments are tied to achieving certain milestones representing a certain amount." And this has been going on for over 100 years.
So what earned value--part of what is happening is people are beginning to realize it's not this nightmare type of situation we have as DOD Instruction 7000.2. It's basically a lot of good sense management and I think it's much more easy to implement now. There's a lot less of this checking all these boxes and stuff.
So I think as you have more and more--Robert's point--as you have more and more success stories in smaller and smaller groups or agencies and so forth, then you're going to get more support.
One other thing. A year-and-a-half ago I was part of a team that investigated earned value management implementation at the Department of Energy. So we were an independent research team who were told to go in and there and savage them if necessary. And at DOE they had--a lot of organizations do this. They had a requirement $20 million and above you were mandated. Below $20 million you weren't mandated. Every one of the smaller projects at DOE, less than $20 million is a smaller project, everyone opted to employ earned value because (1) it wasn't that difficult to implement and (2) they found there was tremendous value.
DOROBEK: Manny, what's your sense? I mean it always strike me that these early adopters, that success if they're successful in implementing earned value that this would sort of get people to go, hey, they're successful and no one wants a failed project. Is it your sense that that will cause--just naturally cause the proliferation of earned value?
DE VERA: Yes. I think in any new concept--but it is really not new but it's new to us in the IT arena--we grab a hold of it and I think what's really happening is as we evolve in a project management discipline, earned value is going to evolve also because I think the two go hand in hand. So even today there's a lot we don't know about implementing earned value because we're learning--agencies are learning and DOD is learning, too, and they created it. I think the idea that it's only for $20 million and above, which is the threshold at GSA, or $10 million at Education or $20 million at other agencies, I think it's a start because those are the priorities right now. But, in reality, new initiatives grab hold at a high level and then as people become educated they realize the value of earned value or management, performance management, in small projects as well.
The best earned value manager that I know of is my wife and she has not taken a course in earned value but she knows how to manage costs and schedules much like the construction industry folks do. So it's being applied. They don't call it--it wasn't called earned value when it started in DOD. It was called cost schedule control system, I think, and now we've given it a name of earned value and there are factions within the project management community where the names and the symbols and the acronyms are changing. PMI calls the acronyms or the factors different than what we do.
So I think we're coming together initially so the die hard early adopters are pulling us along.
Just like the evolution of the PCs and word processors. We didn't know--we had to justify a word processor and a PC when I first came in the Army. We don't do that today. We have policies that you only do earned value for $20 million but I tell the students and practitioners that earned value applies to small projects as well and you don't need a big enterprise license to do earned value. A simple spreadsheet will do. So I think as we become smarter, the two disciplines, earned value and project management, are going to go hand in hand.
DOROBEK: Maybe your wife is taking a class and you just don't know it. Odelia, why don't you talk about EPA a little bit? How is it? What's your experience?
MS. FUNKE: Well, we've adopted it pretty broadly for our business case process. Those two work together in our agency. And what I see is in our really distributed system where we have responsibility and budgets really controlled in a distributed fashion across the agency, I think it could be a very good tool for senior management oversight. So we have a board that comes together and one of the things they look at are those EVM metrics.
One of the issues I hear is that scale-ability has not really been solved yet. So on the one hand it affects small contractors because this still seems pretty formidable and expensive to them and so I think sort of a better understanding of the scale-ability and how it might be used in smaller projects because compared to a place like DOD we don't have huge projects but we do have projects where it can be very useful both, I would say, for the project manager but also for senior management oversight.
And the other thing I would add is I think that it takes training not only of the project managers but of senior managers to understand what they can get from it and the kinds of questions they need to ask depending upon the deviations they're seeing and how it might not be a bad sign if people have, in fact, adjusted their plans because of changing circumstances which they've recognized. So I think sort of automatic reaction that the deviation over 10 percent is a bad thing sort of puts a pall over the whole process that needn't be there.
DOROBEK: All right. Now, everyone has had a chance to speak so we don't all need to be so polite. We can interrupt one another and I've been given a note. So for those playing the home game, we're moving on to question two.
So DOD has been--
ROVINSKY: Let me interrupt you--
DOROBEK: Yes, please do.
ROVINSKY: --since we came--
ROVINSKY: We've talked about it from a requirement point of view. We've talked about it from a good management point of view. I think those are really there but I mean from other things that are really important, I think from the FAA's point of view is our administrator and our chief operating officer of air traffic has made a strong commitment to transparency. I think that is fundamentally more important to me than just tracking a project.
The transparency of being able to have the same data inside the agency with the contractor and with those stakeholders, funders, auditors, that's very important. Everybody looking at the same data. We often get in trouble if people are not seeing the same data. Obviously, there is some data that is proprietary from a contractor point of view. But more importantly this transparency is very important. The ability to show everybody the same data and not to have a lot of energy spent in furrowing it out. As I said, there shouldn't be any secrets.
DOROBEK: It's interesting because I was talking to Admiral Stillman last week of Deep Water and he was saying that he gives--as soon as he has his earned value management data, he gives it to GAO, he gives it to his IG, he gives it to the Hill at the same time that he gets it because that's so important to getting sort of--so everybody knows where all--this is where we are and if a project is sort of not making its metrics that's out there as soon as possible.
ROVINSKY: One of the best practices that we've been studying right now is having a government account officer and a contractor account officer so that each of them are sharing the data on each control account.
DOROBEK: And that sort of is what question two is largely about. What visibility does EVM give you into projects? How much of a sense do you actually have as to where your investments are and where they stand?
BURNS: From my experience at Bureau of Indian Affairs, it has been very helpful. I have a couple of props but standard report. I mean colors, everything. It tells you when you're good and when you're not so good.
DOROBEK: That will look great on my black and white printer.
BURNS: Yes. But what I mean by that is--and I'll give you an example. We had--several of our projects were going very well in terms of meeting our indices of within 10 percent of our measures. We had one project that was under and what it would allow us to do is identify not only was it under in schedule but also under in cost.
So the key factors are where are you in terms of the project moving forward and meeting the schedule but where are you also in terms of the costs. Are you going to have overruns or not? In our analysis in this case we were able to determine that, yes, we are behind schedule but we are also burning at the appropriate rate so that we're not going to affect the overall costs of the program although we will have delay in schedule, and then we can put the appropriate corrective action plan in.
Another aspect of that is it also helps us determine if it was the other case of, well, we're behind schedule and we're burning too fast, and we've got more serious issues that we have to start contending with.
I think what this really gets down to is as we move and the organization matures, and now with more program managers actually being certified, we're moving from, I'd say, the old traditional mandate of you got to report the information to now we're using it to actually manage the project. So it's really transitioning into maturity from reporting to management at this point and that's where it's really a value to demonstrate how we are actually doing across the projects quarter by quarter versus at the end of the year and then trying to explain why we have these either over runs or under runs.
DOROBEK: It's interesting because Odelia mentioned--one of the questions refers to the down side of this and she mentioned the scale-ability issue. Has anyone else experienced that? I mean, I hear a lot about EVM being used for large projects.
REHBERGER: Yes, that is, I think, one of the--Brian talked about maturity. I can't help but notice that EVM is used most successfully where--huge L space on the DOD side, where I mean to us $10 million a year is a huge project, to them it's probably rounding error in some situations. There cost considerations for implementing EVM are not so great as they are in the smaller projects. As a matter, I even jokingly tell people that EVM is one of those unique areas in IT where they are more concerned about scaling down than they are about scaling up because it has been proven it can scale when the numbers are big and the importance is high. Where does that--what's the lower threshold on that? EVM for a million dollar project, full EVM, ANSI-748 compliant, 32 criteria, external review, I'm not sure it's worth it for a million. $5 million? I'm not sure it's worth it there. I think the evidence I have that supports that is that every organization has basically a different set of criteria for when ANSI-748 compliance is required. If everybody would gravitate toward a certain set then that would be the consensus as to what the criteria are.
The other thing in terms of maturity I would like to point out, too, is if you go into Gartner. Gartner is sort of the sort of private sector indicator of where a certain area of understanding is maturing at least in the private sector because they have a very rich dataset in the private sector. If you look for EVM on Gartner, their EVM guidance started last year, which means they have very immature understanding of what EVM is, which means it's also one of those odd things where the where the private sector is lagging the public sector in a lot of regards.
So to me scale-ability is very much an issue. It's very much an issue that is being sorted out on a day-to-day basis. I think OMB was wise to push that out in their memo last year to sort of decentralize that to the agencies to create their own criteria. However, the feedback we're getting recently is, for example, everything has got to be reported--via 300 has got to be using EVM even though that violates our own criteria. So my concern there is where is the downward scale-ability?
DE VERA: The requirement for 300 was for major acquisition and depending on the agency a major acquisition could be $5 million, it could be $50. Okay. So it's not reported on non-majors. So if the agency--if we think it's a big enough project. And I'd like to address the scale-ability. This is real money and all EVM is if you read the 32 criteria is a set of standard good business practices, break your project down into small pieces so you know what it is. If I'm a small business and have an accounting system that collects the data correctly, if you read them they're just basic standard good business practices. They're scale-able because if I have a $5 million project, I don't have as many work breakdown structure elements. I don't have as much to collect.
The other key is we constantly get, "Well, it's expensive for small business." If I give somebody a $20 million project--this is taxpayer money and our history is we're not managing projects at any level. We constantly have requests every year for baseline changes. Some of that is because we change scope but, if we do earned value right, you will start earned value when you do your planning with your work breakdown structure and it will help the government develop the scope and it will help the contractors develop the scope in the proposals.
So--I may be the only one here--earned value doesn't cost anything. It saves us money. It may cost you if you don't have a good management system. We consistently have contractors who can't manage because they don't have a system good enough to know where $20 million is going. So the key is we need to make sure if we're going to spend that money that we have a system that we can track how well they're spending the money.
DOROBEK: Let me go to Manny and then Brian.
DE VERA: I'd like to add to what Dave said because while the criteria in the memo, when you first read it, is scary and the requirements are there and it's staring you in the face, I submit that what good project managers do on their small projects, they're going to do--if they're screwing up those small projects and they're doing it that way, they're not going to change overnight when they get the big projects.
And I think that at least my personal experience using it on $100,000 task order was very simple. It's not that hard. You use a simple spreadsheet and it's much like what the CIOs have done. You get visibility on your 300s but many CIOs wanted visibility of those below the 300 threshold. So that was a management technique that they used so that in case they grew--those small projects grew to big ones, they were ready.
And I think the lesson here is that all project managers should at least start with the notion use earned value to the degree that you would use and apply security. If you need a lot of security on a project, you use it. If you don't need a lot of security, the level of security required, you don't need that on small projects and it's the same thing when you apply earned value. It's just good judgment and project management to use the degree of earned value tools that would apply.
BURNS: I just want to go back to the thresholds. My view is kind of if you're dealing with thresholds you're really talking about earned value reporting and it's more of an immature, yieldy situation. We have to report so, therefore, we'll put a threshold. If you have the more mature organization, any project, in theory, should give an earned value management and it's not about thresholds. It's about gradation of the amount of work that you have to perform to do it. So I think that's what differentiates where you have--where is the cut off? Are you doing it for reporting purposes or management purposes? As you just said, you should really be doing it at any level. It's just a matter of how robust is the analysis that you're doing.
DOROBEK: Go ahead.
ROVINSKY: Let me just add one more thing. We were approached by our chief procurement officer who was concerned about the cost of EVM. Her experience, as many people in the acquisition shop, is people ask a lot of money to do it. It's an add on cost. They didn't see the value, et cetera. We went back and we did a study. The FAA had evaluated all of its major programs as to how well they were using EVM and basically rated them red, yellow or green. And so we basically evaluated our programs with those three categories and then looked at the percentage that they were spending on program management as a percent of total project cost. And we found that those programs that we had rated red, not using much EVM, cost more than those that we had rated yellow. And those that we had rated yellow cost more than those that we rated green.
So we went back and said, "Using earned value management is not adding cost. It's actually reducing the overhead of managing a project." Of course, her response was, "That's obvious." But it wasn't obvious until we did the work and so we're going to actually next week at Clearwater--we're going to actually present those results and I think that's the kind of data that I think is very helpful to be able to go back and show that, indeed, this is not an add on cost. It actually saves money.
WHITE: A few years ago somebody coined the phrase that scale-ability and earned value management and common sense were directly proportionate. I wish I could take credit for that.
MUZIO: I'd like to add that it seems to me that one of the really clear benefits of earned value management is of the idea of articulating performance goals because I've been at the agency a very long time and I think one of the abiding difficulties is people start out on a project without clearly articulating for themselves what the performance goals are. And, of course, what you don't understand, you cannot possibly measure.
And so to me the idea of scale-ability is a task well worth undertaking and I certainly would agree with the comments that, as an approach to managing a project, this really articulates some very fundamental concepts that are valuable for any projects. And I think teaching that to people who are managing projects on the one hand and teaching managers how to understand it and use it are equally important challenges.
DOROBEK: I'm going to toss it to either David or Karen on the scale-ability issue. Do you guys want to just touch on that at all?
FRAME: I think one of the things on the scale-ability is you've got to recognize that different parts of the earned value management system have different levels of difficulty to achieve. Measuring, as I said earlier, how much work has been done. We've got 0/100 rule, 50/50 rule, and about six or seven other approaches. Where on a $100 project, okay, when the task begins 0/100 says you don't put any of the value of the project, meaning the planned value for that task. Once it ends then you go ahead and you close the books. You say, "I did my $100 worth of work."
So in terms of scale-ability, at a minimum, no one should have any compunction about requiring people to be able to report how much of the job have I done.
The problem with the earned value from an implementation point of view is measuring actual cost. Okay. That's a tricky thing and even very sophisticated systems--okay.
I worked with Hughes Aircraft, their electronic--what do you call it? The electronic group out over at LAX airport--a few years ago and talked with their cost account managers who had to maintain the data. And they said, "It usually isn't until two or three months after we've actually done an activity that we're able to figure out what it cost because of things like lags in billing and those kinds of things and so on and so forth." But that shouldn't--so people have got to be aware that the actual cost data, there are--there are certain problems.
I want to point out what Dave Muzio just said about small businesses and whatever, Quick Books allows you to track everything at the past level. Okay. I mean Quick Books is--almost all the small businesses in the United States--you can use Quick Books on small businesses that are $50 million businesses and Quick Books allows you to track the actual cost.
The problem isn't physically the numbers coming out of Quick Books. The problem is, is that actual cost data you're putting in really representing say the end of the activity itself because this money is not reported except in dribs and drabs over time but small businesses can use Quick Books. Okay. So I don't--I think it's a--people are making too big a deal about being excessively burdensome on a smaller operator.
DOROBEK: Karen, do you want to--
MS. RICHEY: Sure. There's a couple of things. Scale-ability usually doesn't apply to GAO because we're only asked to look at big programs that are usually in trouble. But one thing we're doing right now is we're replacing our finANSIal management system and so our management has required EVM reporting during that project. And this project is going to cost no where near the $20 million that you're hearing about. It's going to cost a lot less than that but it's GAO's own budget money so they care about what's happening with the project. Not to mention that there's GAO costs to run parallel systems and all of that.
So we see it as beneficial just on our end to have that in our own internal project.
The other thing I was going to say, too, is about EVM is a lot of people misunderstand it as a cost control. Our experience is that it's very much--cost is definitely one component and it's an important component but the guts of what's making the data come out is actually from a technical background. Variance analysis for cost and schedule always tie back to some physical engineering product or something that's technically based that if you can get that education out there up and down the ranks you can make it more applicable to the engineers. You can make it seem--it's not a finance and accounting problem. It's actually a real technical engineering problem.
And when I worked at Naval Air Systems Command, not one of us was an accountant or a finance or an MBA. We all were industrial engineers, mathematicians or statisticians. So just to give you an idea, we were trained as engineers. That was our category. We were general business industrial specialists is what our code was. So I think too many people run and think it's a cost control system and don't realize the benefits and the richness of the data that you're getting out of that. So I think that's also an education that needs to happen in the civil agencies all the way through the ranks that it covers the vast majority of all the projects that you're looking in.
DOROBEK: Anything else on question two before I let Bob know that we're moving on to question three.
DOROBEK: This one sort of focuses on how you actually use it and, Sally, perhaps we'll start with you. In terms of your agency, how do you actually use this data? How did you start implementing EVM? What does it actually mean?
MS. GOOD-BURTON: Well, I'll talk about how we did it at Housing and Urban Development.
DOROBEK: That would be great.
MS. GOOD-BURTON: Yes. Even though it had a small IT portfolio of only about $100 million, we required EVM reporting on all the projects, all the majors and non-majors. So we had them develop their cost and schedule performance breakdown and submit it monthly and then put it into a portfolio management software and report the data in a portfolio basis to our executives. We had project management review board meetings where we presented to the executives different displays of the information graphics and that allowed them to see what projects were in trouble and how they related to each other. So I think at that department we used it very extensively.
DOROBEK: Bob, does FAA require all projects to go through EVM or how do you decide what projects? You were talking about the guidance that you worked on.
ROVINSKY: At the program level it's majors. Programs that we report through the 300s on to OMB and also projects that come in front of our Joint Resource Council, large major systems. At the contract level, a $10 million contract or more.
DOROBEK: How important is that decision of whether--that threshold question?
ROVINSKY: I think it's important to begin with. I think that's--I think it's an important start. You have to be clear about what you're going to do and who is going to have to do it but we're making it optional and allowing tailoring below those levels. So I think it's going to be a case that when the executives begin to use the data, whether it's through a portfolio approach as Sally mentioned or through another--or coming in front of executive councils--that it's going to spread down. People will see that it's an important technique.
Our director of finance within the air traffic organization has begun using what he calls Quad charts similar to the charts that Brian showed. And all of a sudden all the vice presidents in air traffic are asking what is this data, how is it going to be, and all the program managers are beginning--and you have to--it's management use that is a critical factor. The commitment at the top, some training, but ultimately--and commitment to measure and then finally the executives actually looking at the data and using it to manage. That will make all the difference. It's not the criteria. It's the use.
DOROBEK: John, can you talk a little bit about agriculture and how you have implemented or how the agency has implemented?
REHBERGER: Yes. Again, as I mentioned, it has been done within the context of the project management discipline if you will. And, as such, it's one of the elements we try to push out in terms of our structured training and also it's one of the issues that we have tried to increase the related training for. For example, we have Dennis in all the time to talk to project managers. In our world it's very much a long lead time activity. It's a maturational approach.
As a matter of fact, I'm waiting for the EVM for Dummies book because for the project managers like me that's what I really need. And if you simplify it in that manner then you increase the maturation and you increase the adoption over time.
Pushed by OMB, we created the policy but in our world, to be honest with you, the policy tends to lag the practice. First, we sort of believe and then we write down what we believe. But that's sort of how we're doing it. It's very much a managerial--I mean, a maturational iteration.
DOROBEK: Dennis, are the best practices for implementation how one--if you haven't started--how do you come in there when you're called in to be the pied piper of EVM? What do you say? Where do you start?
WHITE: That's a good question. I think the primary issue in walking into that environment is to understand where their interest is coming from, what's driving it, and then address that first. Then once you get there then you can just start as though you were implementing on a program and literally build it up from nothing to fully operational. Now whether you apply it just to one program or across the board, for instance, the accounting system tends to be a big effort and how you address that simply becomes dependent on their maturity in both. The accounting, for instance, we've encountered some of our civil agency customers who virtually have no accounting system at the local level. That makes it quite difficult to assimilate the actual cost to an activity. So then we even have to start designing ways of accumulating resource accounting.
DOROBEK: I guess I keep coming back to the question if I'm a fed and I'm running a project, why wouldn't I use this? This gives me some kind of safety net that tells me--sort of spells out--it's a process that spells out where I should go. It helps me identify if I'm having problems and yet we all know that it's not the way most projects are managed.
WHITE: That's what I usually ask. Why wouldn't you do this?
DOROBEK: And what do they say?
WHITE: Well, I think it goes back to what some of the original comments were. It's scary. If you ask somebody to take a look at ANSI-748, which is the standard for earned value management--again, we don't have 35 criteria anymore. We only have 32 but they are kinder and gentler. So, I mean, we're making--
DOROBEK: In 25 years, look what we've done.
WHITE: Actually it has been more than 40 years. But I think the point there is there has been some maturity in that. We've actually seen those 35 criteria accompanied by some 130-140 check mark questions go down to where the checklist is gone and there's only 32 criteria now and those are not nearly as directive and prescriptive as they used to be. They are much more broad in context and much easier to accommodate. So the requirement side is changing as well and it is becoming more management oriented versus more contract administrative in nature. I think that's a big part of it.
DE VERA: I think what we're saying, too, and I've heard a couple of folks allude to it, we need to de-mystify this not only with the project managers but with the executives who touch and the stakeholders who touch some of these programs because I think they don't understand it. They want the program manager to get the training but I'm not really sure that they know how to use or are aware that this is a good thing to help them at the executive level because we are starting to use our jargon and EVM and the technocrat jargon. But I think the acquisition community and the CFO community, I think they're coming on board with earned value but what scares me and concerns me is the fact that the executives who make key decisions don't know. So I think something needs to be developed. An earned value for dummy executives or something or something.
But I do know--I met an executive in DOD, the highest ranking civilian in the Army, and she indicated to me, and it caught me by surprise, she said, "I just attended my first presentation on what earned value is." And she said, "I think I can use this." So I think something needs to be done for executives who touch, who fund, who get briefings, who sit on IRBs, just so they get an idea of what it really is.
DOROBEK: David is sitting next to you taking notes about what the title of his next book is going to be.
I think the dummy people may already have that one cornered.
But as you're teaching this stuff and as you're telling people and prostulitizing, a word I never say, but as you're out there talking about it, how do you get people to understand that this is something that can really help you? It's not just something you want to check off. It's not part of a checklist. It's something that, if you incorporate, helps you actually be successful.
FRAME: When you're teaching it what you're doing is you use examples like George and Martha's picnic. I can't remember if that's in the book here. But, I mean, you have--
Yes. That's the new managing projects and organization book.
DOROBEK: Published by?
DE VERA: I was--I used his textbook.
FRAME: Yes, but I'm using you as a--
DE VERA: I just changed textbooks.
FRAME: Oh, okay.
DE VERA: Just kidding.
DR. FRAME: Well, anyhow, you rained on my parade. I think you start with pretty simple examples. You can even actually--it doesn't even have to be--you can have actually task A, B, C, D. Task A is $100. Task B is $200. Task C is $150. Task D is $150. A has begun. This is the 50/50 rule. A is begun so you can say we've done $50 worth of work. Until it's finished you can't say we did $100 of planned work. Okay. B has begun.
So what you do is you just set up a very simple cell and just start saying to people, "Just use common sense. What percentage of the work has been done?" So they calculate the earned value figures. They add them up. The overall budget is supposed to be $800. "We've done $250 worth of work." Okay. So you can basically say, "Okay. Well, we're about 78-80 percent done on the job." They say, "Well, we were supposed to have completed everything." So in the beginning you use those kind of common sense things.
I had--I worked with DSMC, Defense Systems Management College, for many years. I was on the program management program advisory board. The program management program is part of the department of redundancy department. Okay. And while I was there, I would have these students who had gone through a whole sequence of these courses, in a one hour summary using the example I gave you, a lot of them go, "Okay. Finally it all gels." This is after they've been going to this stuff for quite a while.
So part of the thing is to get the--getting back to Dennis' point, let's forget about the 32 criteria for a moment. Okay. Let's--or the 35 criteria with the 128 checklist and all that kind of thing. Let's go back and recognize that this is basic common sense management, which many people have already said. Focus in on that way and then you don't start off with the 32 criteria. Okay. The organization, the planning, this--you don't start off with it. You start off with some common sense and then you can kind of gradually segue into, well, now how do these criteria enable us to use this great common sense tool effectively.
WHITE: There's one other point that I'd like to make along this line. Typically, maybe it's a strong word, but typically we have a tendency to manage based on some dollar figure, our budgets, or we manage our schedules to when something is going to be done. I think one of the biggest benefits in earned value is it forces us to focus on the scope of the work itself and what we are really dealing with is what the content of that work is. So that the cost information, the scheduling information literally become independent--I mean dependent variables based on that scope of work. And I think that's where your prime benefit comes in is you're focusing on what's getting done or not getting done versus where you're at in a budget scale or the time lags necessary. Both very important information but the scope of work, if it doesn't get done, the project is down the tubes anyway.
ROVINSKY: I would say two things. First is that when--if EVM is seen as a technique, a technique that's done in project reporting, I think that it misses a lot of the value. I mean our--we think of it as more like--as a rock that you drop into a pool which spreads out and affects the entire pool. So for our case we're considering it as an EVM transformation. We're looking at many changes in project management, acquisition reform, cost estimating, system engineering, risk management. It affects the entire range of project management in our organization and it basically is a way, as we introduce earned value management, of turning over rocks and finding weaknesses throughout our entire program management area that we're improving.
The second thing is that in terms of education, we have invited in all of our major prime and support contractors to come and tell us about their best practices. One of the questions I ask each one is how do you train your executives? How do you train your new program managers? There are many--many of these companies have extremely rigorous internal training programs that we should adopt. So we're getting their materials. We're making them available. We're going to start adopting them. There is, in fact, an EVM for dummies video but there are many other good management and program management tools. So rather than the federal government trying to reinvent it, the private sector that already has about 40 years experience doing this has good tools that we can adopt.
BURNS: On that point, at BIA one of the things we had to do was come in and in our case starting with no business cases in place and then having to develop them for each one of the programs, I had to establish a program management office, support office, and that office allowed us to give support to the business units to get them educated on how they look at business cases and then from there the next step was earned value management. And wound up hiring approximately eight PMPs, project management professionals, either contractors or feds. And then we were able to take that to the IT investment review board and literally the first major board that we had was a training session. Sitting down with the executives and going through and telling them what this is all about and how this would affect them and that helped a lot because it just sets the stage. Because at that point then as an executive it really came down at the end of the day for the earned value management analysis that we had, three basic things come out of that, it's either continue the project, fix the project or kill the project. So at the end of the day that's what we get, continue, fix or kill is a decision that comes out of this if we do the analysis appropriately.
DOROBEK: Manny mentioned sort of the executive interest in this or lack of interest. Sort of opening it up for folks, what should the executive role in EVM be and do they need to know what this is?
MUZIO: I don't think they need to particularly know--
MUZIO --down at the next bolus. But the easy way to tell an executive what's going on is to show them the EVM data with the red, yellow, green down to the work package level. I went out to NRO and NRO has a Saturday morning meeting with the director and deputy director and the government project manager and the contract project manager, and they start on program whatever, whatever the director or deputy director want. They have eight 42-inch plasma screens around the room and he says, "I want to see all the reds." Whether he knows much more about it than that, he says, "I want to see all the reds." Then he says to the project managers, "Why is it red and what are you doing to fix it and how does this affect the overall project?" Because if it's a critical milestone, the rest of them then are going to say, "It's doing--we're doing pretty well but this one is now starting to slip and, if we don't get it fixed, it affects the time we can put a satellite up."
So from an executive point of view, I think if you just even say okay, let's go in and start with the executive review committee and say, "Here is where they are." And as a project manager and/or an executive, I can tell you exactly where the problem is and then I can tell you why they're having a problem.
And that's really what they need to know is, is this project on schedule and on cost. And while we try to say cost is not all that important, it is absolutely critical. I mean, it's--you have to try to maintain a project within your budget or you can go back through a lot of hoops. I mean, the goal is--
DOROBEK: What's the difference between that and managing to a budget? We said that we don't want to manage to budgets. We want to look at broader issues.
MUZIO: Well, if you don't--part of the decision on building an earned value system is a risk adjusted budget. Okay. The whole key to this is to be able to establish a risk adjusted budget such that when you come to management in the agency, come to OMB, go to congress, and the reason for FACA-5 was people go through the process and they go to congress with phony numbers. Okay.
Every agency--every agency spends every appropriated dollar. So if I gave a low ball on my major project knowing full well I've got to come back next year or in some cases we're seeing agencies that are running out of money in the middle of the year and then coming back expecting to be funded.
And the answer is--the goal is to develop a project which you can achieve within a set of cost, schedule and performance parameters because you've said to management across the government we need this to meet our strategic goals and objectives; we need it by this in order to do something that we need. Maybe it's just to get rid of the old system which is driving us crazy. And will you give me this much budget? And basically I'm promising you that I'll deliver on time this capability within this budget and shouldn't have to go back very often.
DOROBEK: It feeds into that transparency that Bob was talking about.
DOROBEK: Where if you have that transparency all the way through, it makes your numbers--and you have a proven track record of using earned value and it makes the numbers up on the Hill--they're much more likely, it seems, to actually believe them when they come down the pike.
MUZIO: Well, we're not--the problem is that you can start more projects if you use low balls on both sides. Okay. But then you do not allow management to make the right decision of whether to fund the project.
If you know the project is risky and it's going to be $2 billion and you come in through the process and say, "Well, if Murphy is not here and everything works right, it's a billion dollars," you only tell management, "Oh, yeah, we can bring it in at a billion dollars," then they say, "Oh, that's a great bargain."
If they knew that the most likely risk adjusted cost is $2 billion, they make a different decision. They may not. They may say still go ahead and we'll fund it up to $2 billion. That's primarily what they probably would say. Then ask project managers, "What's this going to do to my company? Can I--is it likely to go over the top?"
It's one of the things that Boeing did on the commercial airplane side. Back in the late '60s they complained about the old cost schedule control system on all of their government projects. And I was an Air Force officer going through a year at Boeing and so we went to all the government projects and they all complained about the difficulty of cost schedule control system, that there are five criteria and all of this.
So then we go over to the commercial airplane side of the house and we talked to the 707--they had 707, 727, 737, and then finally 747, and we asked, "How do you manage?" 707, 727, 737 said, "Well, you know, we kind of know how to build airplanes." We got up to the 747 and we asked, "How are you managing this?" And they said, "We have taken the government's cost schedule control system criteria because we can't afford to do what happened on the other three programs because this will break the company." And so they at that point modified it but probably got it pretty close to the 32 criteria rather than the 35. But they essentially said, "We have to do this because if we don't manage 747 within the cost parameters, we are going to put the company at significant risk."
DE VERA: I've got one more comment before I have to leave. I have to reemphasize that what we don't realize, and just to articulate what Dave said--and I've used Dave's capital programming guide also for the past 10 years, and in the back in the appendix it says the purpose of earned value is to make sure that the contractors control systems are adequate. Sometimes we forget that one of the real reasons is really to make sure that the contractor has those systems in place and then if we can do that then I think our COTRs and our project managers will follow suit so it's really a result of making sure that the contractors have those tools to do the job.
DOROBEK: Manny has to jump out in a minute. Do you want to--we're going to finish up asking final thoughts but if you want to give me your final thought as you leave.
DE VERA: Well, my final thought and my hang up, if you will, is I sincerely believe that the three communities, the finANSIal, the acquisition and the IT or project management are getting on board with it that I think that the last piece of that is the executives who touch--I'd go a little further than what Dave said.
I think they need to know more than just the color. I think they need to know a little bit more about earned value so that they can help with the analysis. Perhaps some control charts. They like graphs. Executives and generals like graphs. They can understand the graphs and they're not that difficult. You don't have to have the tabular data and all that data that we've collected but colored graphs tells them the direction of which way the project is going. Above the baseline, below the baseline. They can understand that and if the slope of the curve is going away from the baseline, he ought to be--he or she ought to be able to figure out that it ain't going to ever get to the budgeted cost. And things of that sort that they can figure out and they just need to learn those sorts of things. So if we bring them on board, I think it will help us at our level.
DOROBEK: So the take away from Manny is invest in a color printer.
Thank you for joining us.
Just one other question on this third part that we didn't touch on and that's type of projects. Are the types of projects--and this is probably mostly for agency folks who have been working on this. Are there types of projects that work better with EVM than others?
MS. FUNKE: I think one of the things that--
DOROBEK: You don't have to raise your hand. Just butt in.
MS. FUNKE: Just jump in.
I think one of the transitions that has been going on at our agency is this movement towards performance based contracting and people do seem a lot more comfortable with very detailed kinds of descriptions of what it is and how big it is and what it weighs and so forth.
So I think the move towards performance-based contracting is really in parallel to EVM because if you're going to have more than a costing schedule look through EVM then we're back to the idea that you have to know what those technical performance goals are. And if you had started with a contract that is performance based, I think you have a lot better chance of cross walking what it is you're after with an EVM analysis than if you have a time and materials contract, for example.
So it seems to me that it is an easier kind of fit conceptually if you start with performance based contracting, which appears to be a good direction that people are moving anyway but it's just taking time.
REHBERGER: I have a comment on that, too. The striations that are used out there, the gradations that are used are finANSIal, and I guess I would argue that that is a poor proxy for--again if you haven't heard a theme here--within the general construct of the project management. To me EVM is basically a risk management tool. Right?
So each project has different risks. Where there is high schedule, high cost risk, EVM seems most appropriate. Where there's low cost, low schedule risk--and here I'm thinking big hardware buys, for example--you can spend a million dollars on hardware and it's basically a commodity. You can swap it out. That seems to me--the cost benefit for EVM is not that clear.
Conversely, I can think of another scenario where it's much smaller. For example, some kind of finANSIal management or some kind of budgeting or some kind of upward reporting system that is not high value but may have--be high profile and maybe high risk. There even if it's small, it's worth doing EVM on. So EVM, to me, again, is a risk management tool within the context of good project management.
So, therefore, I'm not sure that the dollar thresholds are all that useful. They provide some guidance and I think they provide a starting point but in terms of how to implement it, I'm not sure they are not useful.
DOROBEK: So you would measure it rather in terms of risk?
REHBERGER: Well, I think--
DOROBEK: Is that a better assessment?
REHBERGER: Well, I think it's a risk management technique and there are other risk management techniques.
REHBERGER: And that--and that's why projects are supposed to be unique initiatives and so you have to look at it and you need--you need construct and then getting back to Dave's comments about basically what they did is Boeing--the company said, "Well, we can afford this to fail and we can afford this to fail. We can't afford this to fail. Therefore, this is high risk and we're going to go full C/SCSC." That's what I heard.
And so to me that's more of a risk management concept than an EVM is good concept. So, therefore, I would say EVM as a risk management tool--you have to look at a risk management tool, not a EVM is good, all the EVM technique.
MUZIO: Remember that the guidance that we have on systems is--that's not at a dollar threshold. When we first started in 1996 trying to write the capital programming guide for the definition of what a major acquisition is, the general concept coming out of DOD was earned value doesn't apply below $20 million and the initial concepts to management was let's just make it at $20 million. It's easy. It's an easy way to just say, "Okay. $20 million and above, you've got to do it. And below that, you don't have to."
The definition of a major acquisition is high risk, high visibility and importance to the agency mission. It does not have a dollar threshold. We're constantly asked by the agencies, "Why don't you give us a dollar threshold?" And that is exactly--the reason is some things that are $1-2 million are very important and need to be controlled very carefully, and others are not.
Now agencies tend to just put a dollar threshold because it's easy. That's why the guide also says that--okay, we'll accept the $20 million or whatever in the normal process but OMB can ask for any project to be declared a major. And we would only do that when we think there's some reason that the smaller project is very important. So we would say we want earned value on this project because it's an integral part of our ability to make a budget decision or your ability to deliver to the taxpayer something. So we want to see that small project and then once it becomes a major acquisition under 300 we get to see it for the rest of its life.
DOROBEK: Lucky you.
Anything else on this issue before we move on to question four? No. Good.
This one talks a little bit about particularly those high risk and projects that we always talk about projects that have failed, and I think there are probably not all that many of these systems that we just decide that we're going to close down and move on. How does--can anyone talk about instances where EVM has either helped them either save a project or decide to close down a project? Brian is nodding his head so I'm going to go to him first.
BURNS: As I mentioned earlier, being an executive over a lot of these things it has come down to the red and green view of the charts. It has forced me to the issue of continue, fix or kill.
In one situation where I mentioned we had a project that was behind schedule but within the--also behind in the cost, that project was fixable in the sense that we identified pretty quickly what the issues were and we took appropriate action. In this particular case it also had some contractual issues that we had to deal with and we were able to go back to procurement and work some of those things out. So it fixed the program at the appropriate time before it got to the point of having to make the next time a kill decision because the schedule was behind and the cost was also overrun at that point.
DOROBEK: Just talk to me a little bit about--for people who may be reading this and aren't as aware of what--how EVM works, how does it tell you--how do you find out where in this whole--all of the potential issues, all the potential pitfalls that you can fall into, how do you find out--how do you know that it was a contracting issue that was a big issue?
BURNS: Well, first of all, we did our own earned value analysis and we came out with a number. So that first of all gave us an index so to speak. Are we ahead or behind or on target? From there, we were then able to say, "Okay. Well, we've got an issue here."
So the next thing we did is we then--it's like going into the store. You see the window. The window is colored red. Now we're going to go into that store and look what's behind that window, and that's what we did. Then we looked and said, "Okay. What are the issues? Why are we behind schedule?" Well, we've got some contractual issues here and we've got some technical issues or some other things.
So we just went to the next level down and started looking at the milestones on the schedule. Why are they behind? And then what's behind those milestones? From there to the actual work package or work breakdown structure of what had to be done. And that helped us determine why we are behind schedule and what the actual issues were at that point in time. So that's how we were able to use it and actually salvage that at that point.
ROVINSKY: I want to talk about a particular instance in our agency. We had a multi-million dollar critical program that had fallen behind in schedule. Not only that but it got worse over the last fiscal year, and of course that caught the attention of the department and it caught the attention of the agency and it caught the attention of OMB and the program manager, and had to appear before three boards. At that time the program manager brought in--this was a green program, a program that had full earned value management in there.
He brought in his charts, his level one and level two charts, and showed where the problems were both with the prime, with the support contracts, and with the government implementation because we--he had done full earned value management on the FTE portion as well. And, as a result, he had a very clear story to tell. There were no secrets and people could understand exactly what he could do and exactly what he could not do.
He had the worst schedule variance of any of the programs we had and yet his experience was basically--after that all three review boards told him, "Thumbs up. You're fine." They did not affect his money. They basically told him to re-baseline and fix--understand the problem and what the issues were. The programs that had far less variance that we had rated red and they went in there and they couldn't explain some of the problems. In the future--the executive board then told this program manager of our communications program, they said, "In the future just send your charts. Don't even come back. Just tell us how you're doing."
And that's the kind of message we tell people internally and give some credit to both the department and OMB because they could have punished somebody for telling the truth but they actually didn't. Variances are a fact of life. They happen. If you have no variances there is something wrong. There's a secret. And so variances are real. Variances are understood at a certain level.
We have to send that message back. This program manager got a very positive message back for using EVM. As I tell people, "When there's no secret, there's no story. People are not going to paying a lot of attention when everything is clear."
DOROBEK: There's that color printer again.
DOROBEK: We'll go to Dennis and then we'll come over to Karen.
WHITE: Let me just jump back in. Using earned value is not the only piece. Using it correctly is probably just as important and the reason I'm saying that is the A12 program, the Navy A12 program, was one example where it was being utilized. The warning signs were there. The analysts were trying to make it known but for some reason the action wasn't being taken early enough and, by the time the action was taken, it was too late and the program got cancelled.
Who knows whether that could have been saved had the information been used properly in the first place. We don't know and none of us can go back through a time machine. But the indications are if you start taking correction action early, you can correct problems. If you wait too long, the problems become unsolvable.
MS. RICHEY: That was my exact--I was going to bring up the A12 because when I started at NAVAIR it was December 31st, 1990, and the A12 blew up about April 23. And so I was at NAVAIR when that went down and as a result it was very embarrassing for the service to have that happen and they--
DOROBEK: It had to be loads of fun, too.
MS. RICHEY: Well, it was pretty bad because there was all kinds of politics, too. I mean, our shop knew--
DOROBEK: I can't imagine.
MS. RICHEY: --that--they knew it was going to be a billion dollar overrun on a firm fixed price program and that there was just no way the contractors could absorb that type of a loss. But the benefit of that actually is that NAVAIR now is a powerhouse agency in EVM. They have a centralized cost community. Within that cost community are more than 70 EVM analysts with people specially trained in critical path methodology and schedule analysis, and that exists no where else in the federal government. You don't have it at NAVSEA, which actually has to do with lots of shipbuilding, and there you're talking about billions of dollars for each ship being built. It doesn't exist at the Air Force. It is no where in the civilian agencies but NAVAIR learned their lesson and they now are a model. They're running around teaching other people, going to conferences, all of that.
So that huge mistake, the A12 program disaster, turned out to be the best thing that ever happened to NAVAIR. At the DOD cost analysis symposium a couple of years ago, there was a presentation where they looked at the F18 program compared to the F22, the Navy versus Air Force, and the Navy showed that because EVM was working well on that program, they were on cost and on schedule without any re-baselines. In contrast, the Air Force program, it had a lot of cost delays, cost overruns, schedule delays, lots of problems.
So I think EVM became--they embraced it. They bought into it. They trained everybody. They hired in the personnel and that's something that civil agencies need to think about. There is no DCMA for a civil agency. There is no independent cost agency like the CAG for DOD. This is a problem because if you--you're going to have more A12s if you go with unrealistic baselines to begin with.
If you go ahead forward with a budget and say, "I can do this," and it's a buy in scenario where let's just get the budget ledge and get the program started, we'll worry about the money later, EVM can actually give you data, an early warning that you're not going to--you're never going to make it. That program was doomed from day one because it just has too much risk in it.
So we at GAO love to use EVM data and love to get it to be able to see that because early on we can tell and we can present it to our management at GAO and we can give it to the decision makers at congress to let them know ahead of time.
DOROBEK: Let's continue on with this sort of example or maybe not use this example but an example like this one where they are using EVM and still you're saying this is high risk and you know it's high risk and you have a lot of--at what point do you say, oh, my god, this is off track? I mean, they knew that they were having--they had red flags. At what point do they say, oh, my goodness, we need to quit or this is just not do-able at this time or how early in that process do you start?
MS. RICHEY: I mean you can start as early as you want. We have been looking at the Missile Defense Agency and there was the ground based mid-course defense program. That has been a huge problem, too. I mean, it has cost a lot more than what was originally promised from the get go.
Now they've also had the boost from September 11th and the risk of being attacked and that you need a missile shield but it happens all the time.
And even at DOD where they know better on how to do this, they have a lot of experience at Boeing, Lockheed, Northrup Grumman, all know how to do this. They still had problems on the GMD program and were trying to somewhat cover it up. They were trying to say, "We're going to apply management reserve to offset this overrun." And DCMA said, "That's a direct violation of your own system description and you can't do that."
So there were issues with that so it still happens.
Everybody is under extreme budget pressures. Everybody needs to get their programs funded and EVM--we see EVM as a great tool to try and uncover that because it is--it's happening in every one of the services. Civil agencies probably too. We just don't have the data. We don't have a lot of the EVM data to see that.
But the benefit to us is again it gives accountability. It very much exposes the problems. You can get down to the problems and root cause to analyze very quickly. The best you can do is raise that information up and give it to decision makers and let them decide whether or not to continue the program but at least you can do your job of giving them objective data to make that decision.
DOROBEK: Let me play devil's advocate for a minute and then I'll open this up for anyone. I can pretty much go through a lot of agencies and I don't have any EVM data but I can pretty tell you which programs in these agencies are probably going to have trouble or are having trouble. What do I need this--what do I need this whole framework of EVM for?
FRAME: Chris, what it does is--I think Karen hit on it and other people have--is it gives you very early warning. Very early warning. So like--what was it--NORAD back in the 1950's and '60s for our defense system. Attack over the North Pole kind of thing and so on. So it gives you very early warning.
But a couple of things that Karen mentioned and Bob mentioned before that I think as we are wrapping up here because we've been a cheering squad now for earned value, I want to talk momentarily about the shadow side of earned value. Okay. Because, as I mentioned, I was doing some work for the Department of Energy investigating their systems, I've worked with other groups and so forth, at the very opening salvo of comments somebody pointed out that the system is not that tough to game it. Okay.
: That's my other book by the way.
FRAME: Yes, the gaming. Well, let me tell you the best--the most comprehensive work you've ever going to see on earned value is written by Quentin Fleming and he has a whole chapter in one of his earlier books called Cost Schedule Control Systems Criteria. It's a very exciting title. Almost made it to the New York Times list of things not to read. Okay.
DOROBEK: Surprisingly enough, it made my list.
FRAME: He has a whole chapter on gaming the system. The front end loading, the back end loading. And you can see all the contractors saying, "This is the best chapter in the whole book," and all that.
FRAME: Highlighted, right. Making illegal photocopies and all that. But the point is, though, for example, this is why it gets back--I can't remember, Karen, if you said it--but you said using--having--bringing on board experienced earned value people. Okay. I'm sure Dave Muzio can look at one of these charts and everything is so on target, and right away something stinks here. Okay. Something really smells bad and all that.
For example, my background is a statistician. I'm not proud of it by the way. Okay.
: I'm proud of it.
DR. FRAME: Don't be proud of it. Okay.
DOROBEK: With statisticians around the table?
[Laughter and simultaneous discussion.]
DR. FRAME: But the thing is statistically if half of the stuff at the work package level--people reading this--at the lower levels of detail, if half of them are grossly negative and half of them are grossly positive, what's going to happen with cumulative curve? Right on target. Okay. The other thing is part of earned value that we haven't talked about is the re-baselining process. Okay.
And at one of the agencies I worked with we had these astonishing success stories of the two or the three lines, the earned value line, the actual cost line, the plan cost line, and all that all being right on top of each other and you are going this is a miracle. Then you talk to people actually on the team and they're going this is a nightmare. Okay.
But you can do--I guess what Dave suggested is rather than--as an external investigator, and I see this really suspicious looking stuff, then when you start going down, and then you start asking for data--this requires an experienced person. You ask for--give me more level of detail. Then you see 50 negatives over here, 50 positives over here, you add them up and they wind up being zero. Okay. So right on target.
MS. RICHEY: Like for us, we've seen actual--mostly from civil agencies because they don't have a lot of experience yet in doing this, you'll see things like a negative EAC, which makes no sense.
MS. RICHEY: Even though there has been millions of dollars in the ACWP recorded, the EAC is negative and then some are positive so even there you end up with no variance at completion and just blatant mistakes that if you run and look at the bottom line, the bottom line may look okay. It's when you go back and look at level two--level two, three, four, and you start seeing these anomalies jumping out. That makes you under--makes you question the validity of the system and what's really going on. Did people really even have a clue what's happening? And that has been very helpful for us, too, to go in and uncover that.
FRAME: I think again you hit the key word, "validity of the system." I think the single greatest innovation at the Defense Department since George Washington and the procurement--they did a lousy job in those days--is independent verification and validation. So somehow in this process--you said, "Validity." Okay. Is this really valid? Somehow--now this is the dirty--this is the part that's scary about earned value and all that but somewhere periodically the whole systems have to be validated but also on a project level you need outsiders coming in and taking a peek, really experienced people. Dennis is the kind of guy who is--this is part of his job. No, I'm saying he comes in and you're going this stinks to high heaven or this is--when you go down a level you basically say it's okay. You're talking about literally--in doing that kind of investigation in earned value you can--literally an experienced person in a couple of hours going through some of the books will see these numbers just don't make sense. How can you have a negative estimate of complete?
MS. RICHEY: Exactly. And we also saw like on one of the shipbuilding programs we were looking at, for a billion dollar ship, the form F5 was one page. And so even the analyst said, "We don't think anybody reads this. We don't think anybody at the shipbuilder reads it. We don't think anybody at the Navy is reading it. Nobody seems to be reading it." And that program had a lot of problems. It had cost schedule variances. There should have been pages and pages of work package explanations of what was going on and yet it was meaningless. There was nothing in it. And that again is coming from an agency that has had 30 plus years experience doing this.
So our concern--I mean we're very happy that EVM is being pushed down to the civil agencies and programs have to do it but our concern is you could check the box and say, 'Hooray, we did EVM,' but it's meaningless. It doesn't mean anything. Nobody knows what the report means. Nobody knows what's going into it. They all think it's great because they don't know any better. And so that's one of our jobs at GAO is trying to train people. Don't just take it with a grain of salt. You've got to go in. You've got to be able to look at this. The reliability of the data. We rely on system validation reports. And then IBR to show what risks or what action items are open. Those are things that we arm the auditor with that just because you get these reports doesn't mean that they met the goal. You have to actually see if they mean something.
MUZIO: We understand you can game it. You can game anything, any kind of a system. In earned value, though, there's always a catch point. The day you come and say, oops, yesterday I told you--and I think it was Murrell [ph] in FBI that testified one week that everything was perfect and the next week they had a $300 million overrun.
FRAME: That was a very bad week.
MUZIO: But that is the thing is that you can game any system but in this system you will end up coming back and saying, "I have been lying to you for the last three years because now I need $300 million." Well, how come that just showed up yesterday? So what you hope to say to people is the system tells you early where your problems are, you need to go after them, that will reduce your overruns if you start early, and don't lie to us.
The Karen memo last year did what? Because even at our level we realized that 85 percent of the programs were baseline change. And what did she say? We could tell that the baselines in the agencies were no good. So what OMB said, "You all must do an IBMB [ph] or an IBR [ph] on every baseline on every major acquisition and report to OMB what you're doing because baselines--we could tell that the baselines were no good." So you don't catch it always as soon as you should but you catch it.
WHITE: There's a couple of points that I think affects all of these comments. First of all, the EVM discipline itself takes into account all of these things that have been mentioned. The idea, though, is if you don't follow them they don't get done or if you follow them in an incorrect way you end up gaming the system. So the discipline is at the heart of that. Dave mentioned the risk aspect.
Risk management has been a part of earned value from the '60s. It was written into the early guidance to utilize risk mitigation. These programs that overrun tremendously are typically because of tech risk--I'm using the wrong term but the technical risk associated with getting the work done. They're trying to build things that haven't been invented yet. And not putting enough risk and risk planning into the front end of that program results in these massive overruns. If people did the risk adjusting up front or put the risk mitigation plans in up front so that we'd have that early on, on the risk side, we would have even better early warning than what the EVM by itself would provide.
So I think those two things kind of feed into this whole discussion.
DOROBEK: We're rapidly running out of time so I want to jump past the other questions and sort of go to some closing thoughts. I have to tell my statistics--I only know one statistics joke and that's 78.4 percent of all statistics are made up on the spot.
I thought we'd just go around the table and sort of give closing thoughts and, if you don't have any particular closing thoughts, perhaps we can just talk a little bit about sort of lessons learned from EVM, best practices, what--if you had to tell somebody who is approaching this right now, what would you say to them? And we're going to start sort of--rather than starting from me, we're going to start over there with Dennis.
WHITE: I think one of the first things that typically we do when we're trying to help somebody get going is sort of the gap analysis where you walk in and look at what processes they have in place versus what ones they don't. The thing that is missing most of the time is any kind of project management. They are very operationally oriented or very functionally oriented, not very project oriented. So one of the first things that we kind of end up doing is getting into the PMP side of project management and getting people to understand what that means and what that is.
So I think the major lesson learned that I've gotten over a number of years now is that you've got to have the program management or the project management concepts behind you before you start working on this. Otherwise you're learning an entire environment, which is what typically happens.
MUZIO: I think we have to--the basic thing is for the government to understand the customer is not the agency. The customer is the taxpayer and that when they go into these projects they're basically going through the system and they've promised management that I can deliver this program which meets the strategic goal and objective need in this time frame, which will save us money because we can shut down for this amount of money. So they've made a promise and yet most of the promises when they go into them aren't real promises. They're just getting into it to get the money and I think we have to change the mind set to have agencies understand and put the discipline in to develop the goals that are risk adjusted and are realistic so that management can say yes or no, and they can actually deliver what they promised to do to make this agency a better server to the public and that's not a mind set that they have. It's I've overrun, send some more money, and we need to change that mind set.
ROVINSKY: Since our audiences are really agencies that are starting to think about earned value management that have received the memos and received the guidance, I think we should--there are a couple of things that I basically would address there kind of as lessons learned in doing this. The first is, I think, as we've said before, is it's not--there's no value in continuing to game the system or lying. I mean, Karen's group at GAO is developing a lot of expertise to come in and review things. OMB has expertise. There is a lot of expertise out there that's going to find out if you are gaming the system. So it's not going to work just as trying to game the Exhibit 300s did not work a few years ago, number one.
Number two is bringing good people to work. Just with any other initiative of the federal government, there's going to be a lot of mom and pop shops who suddenly discover earned value management and they're going to come in and tell you they do it. They'll arrive with checklists and they'll go through checking things off. It's very important that whoever you bring in to help you be people who can work directly with program managers, who have empathy, who preferably have been program managers or who have worked with them. I think that's extremely important. Bringing in the classic consultants--we know who they are--to just come in with some academic knowledge is not going to move you in the right direction.
The third message is also work with your contractors. Many of these contractors are ahead of us. They have had the experience. They have best practices to teach us. They don't want failures any more than you do. So the fact that the contractors are in there just to make the buck, I think, is a mistake because it's a terrible blot on the contractor's record if they fail and if you fail. So they have a lot to teach us. They will be happy to give you the data, I think, and be happy to work with you. I really think that's the case that we're finding.
And the last thing I want to mention is that there is a community of interest that has begun across the federal space sponsored by GSA called the COI, the community of interest, which is meeting all across the government now and bringing in best practices and sharing experiences. I certainly urge every federal agency and everyone who is doing this to participate in this COI because it's a wonderful opportunity to learn from each other. They are inviting contractors to come as well if they're brought by their federal sector.
DOROBEK: Give me contact info and we'll attach to that along when we--
ROVINSKY: George Leow [ph] would be the person to contact at GSA and I'll send you--I think I already sent you the information.
DOROBEK: Perfect. Sally?
MS. GOOD-BURTON: Well, I think the key aspect of EVM is that it forces people to do good planning. I mean, instead of rushing ahead and trying to come up with a finished product they are forced to be realistic in their scope and define the realistic schedules and develop that baseline so that when--and baselines don't have to be rigid forever. They can change if there's legitimate reasons to make a change but that discipline enables good project management and it is a risk management tool as John was pointing out.
DOROBEK: Karen, I'm going to ask you to just keep it to like 40 seconds because we have a couple of other people and it's 10:00 o'clock now, and we promised.
MS. RICHEY: Okay. All right. Really quick. We are embracing EVM. We actually are putting out federal guidance. It's a cost assessment guide that we're working on right now. It's going to take earned--cost estimating and best practices, leading practices in cost estimating, and what puts together a credible cost estimate and we'll link it to EVM to say you do an estimate, it has got risk in it, it gets awarded into a contract and you monitor the progress of it in a contract using EVM. And we want to be able to alert the auditor on what to look for in an EVM system to see if it is valid, if it is getting good data. If it passes that test, then what to do with the data. How to take it to do independent assessments, how to do the drill downs for risk analysis, looking through the Format 5's, taking what kind of variances you're finding in the EVM system and linking it back to the risks that were identified in the original cost estimate.
So this guide is something we're in the process of developing. We are opening it up to anyone who has interest to help us develop it. We have a community of experts right now, mostly DOD so we're trying to more civil agencies in, and Keith is definitely aware of what we're doing.
So if anybody is interested, get in contact with me. We have meetings every other month at GAO. We talk about different topics. We're getting comments from people as we develop the guide and we will have--eventually the product will be an auditor checklist that will do an actual grade--a score grade of maturity and reportable findings, which are things that aren't--you can't nicely fit into a checklist objectively like, for instance, what's the EVM data telling you type thing.
So that's something we're in the process. We're trying to get it out in the fall.
DOROBEK: How soon do you anticipate? The fall?
MS. RICHEY: Yes, late fall. So if anybody is interested, please get in touch with me. I'll give you all the information that we have to date and then let you know what--our next meeting is June 25th.
DOROBEK: Great. Brian?
BURNS: Okay. You mentioned earlier about what's the difference between budget and EVM. Budget in my mind is a one dimensional issue of what's spent and what's not spent, and the outcomes of that. Where really EVM is the proverbial three legged stool between scope, cost and schedule. And if any one leg is missing it's very unstable so you have to be balANSIng them.
One legitimate reason for re-baselining is sometimes with a scope change versus scope creep. There's a little bit of difference there. But often what I have found is in the re-baselining the root causes, quite frankly, are usually requirements weren't fully defined, project management hasn't been fully engaged, or the users have not been involved in the test plans and sign off on the test plans, and the actual test results, and that skews the schedule.
So basically what EVM is about is project management and risk management, and then it gives us basically a measuring stick or a common criteria that the organization, OMB and even congress can use across agencies to see how they're doing.
DOROBEK: Great. David?
FRAME: All I want to say as a statistician that I agree with 74.8 percent and what my distinguished colleagues have said so I don't need to say anything more.
DOROBEK: Excellent. John?
REHBERGER: I guess I would summarize and say that you should look at EVM within a constructive good project management. If you do so, project management will last. If you don't, if you view EVM as a silver bullet panacea, it's going to fade away like TQM or MBO or any other three letter acronyms. As such, I was hoping to get to question five and that is how come EVM doesn't measure quality. Well, because it's not supposed to. The quality process is supposed to measure quality.
FRAME: John, EVM is a three letter acronym.
REHBERGER: I know.
MS. FUNKE: I think my last words or take away is just a greater optimism that EVM can be used flexibly and should be to manage not only the project scope but also the change in scope, which I think is an important thing because it's not just a problem in the government. It's definitely a problem in the private sector as well, the creep that sometimes can overtake projects.
And, secondly, that we really probably need to pay more attention to executive education about what this tool is, what it can do and how to use it to best advantage.
DOROBEK: I want to thank you all very much. I think for many, particularly on the civilian side, you're just really starting to get into this and I think this will be a good sort of primer for them and help them understand why it's important. As a journalist, we like openness and everything. I'm ecstatic to hear people saying we're sharing this data and I think that's really important in terms of cost estimates and all those kinds of things.
So I want to thank you all very much. I want to thank the AFFIRM folks. I think they did a phenomenal job.