Kelman: An embarrassing question
- By Steve Kelman
- Jun 06, 2008
During a recent trip to Brazil to speak at a conference on public management, I was contacted by a Brazilian public administration graduate student with a straightforward question: Could I direct him to data sources on the performance of the U.S. procurement system?
In speaking about the world’s richest country, I was forced to give an embarrassing answer: We collect surprisingly little such data. As a longtime fan of using performance measurement to improve government performance, I think this is a real shame.
We do collect some kinds of performance data. The government tracks cost growth on major weapons systems. However, cost growth is often a function of overly optimistic initial cost estimates. The degree of optimism varies over time, depending on how tight budgets are and how tight the competition is to get programs funded. Those factors make it difficult to get useful data for comparing the relative cost of major weapons programs at different times. Inspectors general sometimes audit spare parts prices, but they typically do so to find outrageous examples rather than to examine overall trends.
Collecting good performance data, particularly trend data, on the procurement system is often difficult. Particularly for services, many contracts are unique and therefore hard to compare with those that come before or after.
Nonetheless, a procurement system that took performance measurement seriously would have no shortage of valuable data that could be collected. Just to cite two examples of low-hanging fruit:
- Prices paid for IT hardware and commercial software, compared by agency and contract and with commercial retail prices. It is shocking that we don’t have historical data on this. We should be collecting data on the Top 10 agency information technology commodity purchases by category to learn how the government is doing and which contracts are delivering the best prices.
- Customer satisfaction with vendor performance on randomly selected (stratified by dollar value) of IT services contracts. In 2004, two of my students examined customer satisfaction on 100 recent IT services purchases from schedules and found average satisfaction of 9.3 on a 10-point scale, compared with only 6.8 in the late 1980s based on research I did for a book before the procurement reforms of the 1990s occurred. It would be interesting to see if this satisfaction level has gone down since 2004. Satisfaction should also be tracked by contract and agency. Make the surveys simple — just ask customers to rate overall vendor performance between 1 and 10.
Can we justify spending money to collect this type of data when the contracting workforce is so stretched? Companies, which typically put more resources into collecting performance information than government does, find that such data collection more than pays for itself in improved performance.
Beyond that, if IGs redirected resources from endless audits of compliance with process rules to audits of procurement results, good government would get a double boost: Gathering performance information is worthwhile, and process compliance audits often are not. Given the importance of contracting, our failure to gather performance information is a scandal. Kelman
) is professor of public management at Harvard University’s Kennedy School of Government and former administrator of the Office of Federal Procurement Policy.