TheLectern

By Steve Kelman

Blog archive

Why a GAO report documenting improvement is not good news

man graphs performance

The Government Accountability Office recently issued a new report called "DOD Acts to Improve the Reporting of Past Performance Information", documenting progress the Defense Department has made getting past-performance report cards on contractors submitted into DOD's past-performance database on time. Now 74 percent of reports are being submitted on time, compared to 56 percent two years ago.

Sounds like a good news story, right?

I'm not so sure.

Done right – and the government currently is a long ways from doing it right, unfortunately – past-performance report cards are one of the most powerful tools in the government's toolkit to incentivize better vendor performance. Everybody knows that considering the past performance of people from whom we buy is one of the most powerful tools customers have to make the market system work better – suppliers treat us right in the hope of getting our repeat business.

There are serious problems with the current operation of the past-performance system in the government, as I have frequently written. The biggest problem is insufficient honesty in reports, so they don't differentiate enough between good performance that should be rewarded and bad performance that should be punished. I have frequently argued that the government should eliminate the contractor's ability to challenge a past performance evaluation they don't like, because it disincentivizes government honesty. Putting their own version of events into the system should be all contractors are entitled to.

Homeowners who have used the Angie's List service will know what I'm talking about. People who have had experiences with service companies post reviews about them on the website. The companies may post responses to the reviews, but have no power to have them changed or removed.

There is also an attitude by many, including the punishment-first caucus on the Hill and some so-called "watchdog" groups, that past performance should be used only to punish the bad, not to reward the good.

Unfortunately, the GAO report -- despite what I am sure are good intentions, and a title that sounds promising -- doesn't really help. In fact, it may hurt the important effort to improve the government's past-performance system.

Why?

What the GAO report discusses is the extent of compliance with the requirement to submit past-performance report cards, and they praise DOD because “compliance” has increased. The report repeatedly uses the word "compliance" with regard to past-performance report cards, sometimes several times in one paragraph. But when government people hear the word "compliance," most think about something unpleasant but (probably or at least possibly) necessary. It's not something we want to do, but a bitter pill that must be swallowed for the greater good.

The second problem is the performance measure the report uses to measure compliance. Although GAO itself has in the past complained about inadequate content in the report cards that do appear in the system, in particular with regard to descriptive narrative (they should also complain about the lack of differentiation in grades), there is none of that in this report. All they measure is the input of how many reports get submitted on time, not about whether they are any good or whether anyone is using them.

Between the emphasis on compliance and the use of a poor, input-style performance measure, the GAO report sends a dismal signal about what past performance report cards are about – they are a "drill," a "data call," imposed by "them" for their purposes, not for us on the front lines of the contracting system. If we fill in these report cards, the only reason is to tick a box. This is not for us, it is for them.

But the past-performance system absolutely should be for "us" – for the customers of the contracting system -- as a tool for getting better contracting performance on our contracts. Clearly, there will be a "free rider" problem – even in a system with honest, valuable report cards, most users of the procurement system would prefer that everyone else fill out their report cards while neglecting to complete their own. A procurement system that took past performance seriously would look for ways to deal with the free rider problem, perhaps with a feedback system similar to that consulting firms use for the lessons-learned reports consultants file for use by other consultants, perhaps including some kind of rewards for the reports that the most users say influenced their contract award decision.

My point is that we need some serious thinking (and action) about improving the way the past-performance system works. I don't think this GAO report helps.

A job for OFPP?

Posted on Jul 11, 2013 at 2:10 PM


Reader comments

Tue, Aug 6, 2013 John Inman Williston, Vermont

We need to remember the basics, and it is a basic and fundamental truth that past performance is an extremely important factor in making a purchase decision, especially an important purchase decision. In this context, past performance = reputation, or past performance = confidence in a successful future outcome (based on the past). Government contracting officers and source selection officials need to make sound purchase decisions, and sound purchase decisions MUST include past performance -- at least, that's how I tend to see it.

Fri, Jul 12, 2013 Steve Kelman

Thanks for these comments from blog readers! Al, as a general matter, if something doesn't work, we should look to trying to improve or to scrap. I would be surprised if there were no connection between actual past performance and actual future performance but if, hypothetically, this were true, yes, wouldn't make sense to devote resources to this. Note that most of us in everyday buying decisions do go back to the same auto repair shop if we believe they did a good job last time.

Fri, Jul 12, 2013 Mike L.

We have been seeing more and more RFP evaluation criteria that read, "in the absence of past project references or performance reports, the bidder's past performance will be deemed acceptable." Pretty much defeats the whole purpose!

Thu, Jul 11, 2013 Molly G

While past performance is always listed as an evaluation factor, no matter how many times they say things like "best value" or "technical is significantly more important than price," in practice it always comes down to low price.

Thu, Jul 11, 2013 Al

A compromise for immunity on the content of those past performance reports would be to concede that any data over 5 years old is purged. Very old data should be discounted anyways (relevancy and *currency*), so this should not be a huge concession. This may also dissuade litigation based upon contested information in the database. I know you are a big picture-type thinker, Steve. What if we study past performance data and find very little correlation between past and future performance by firms? Should we scrap the past performance regime entirely in that case?

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above