Kelman: Let the data talk

More than data, anecdotes inform the debate about recent federal acquisition changes

Editor's note: This story was updated at 3:30 p.m. July 10, 2007. Please go to Corrections & Clarifications to see what changed.

Those who want to go back to the procurement methods of the 1980s believe subsequent reforms have led to rapacious contractors getting huge government giveaways. Because of defense spending’s visibility and its ties to the war in Iraq, much of the reform debate is about runaway increases in weapons costs.

I’ll explain, but first let me repeat an idea that is common wisdom among social scientists: The plural of anecdote is data.

Two studies offer data pertinent to the debate. The studies focus on the effect of procurement changes of the mid-to-late 1980s and acquisition reform in the ’90s on the growing weapons costs. Both look at weapons contracts completed in periods before and after the changes of the ’80s and ’90s. The studies examine several hundred contracts.

One study by Christensen, Searle and Vickery, published in 1999 in the Acquisition Review Quarterly, looks at the first period. That was the time of the Project on Government Oversight (POGO) holds up as the last era of true procurement reform. But actually it was a time when competition rules were bureaucratic, laws made it harder for government and industry to communicate, oversight was the watchword, and bid protests flourished.

The second study, a 2002 master’s thesis on cost analysis by Andrew Mosier, looks at the effect of acquisition reforms from the ’90s to the end of 2001.

Here’s how I would summarize the two studies’ findings: The so-called reforms of the mid-to-late ’80s produced an increase in the average cost growth across all weapons systems compared with the previous period. By contrast, the procurement reforms of the ’90s produced a decrease in average cost growth compared with the previous period.

The first study compared the growth of contract costs completed between 1988 and 1991 with those completed between 1992 and 1995. Given the time it takes for a weapons contract to be completed, one can surmise that contracts completed between 1988 and 1991 were undertaken before the reforms of the ’80s were implemented.

The study of the ’90s took a similar approach to contracts completed between 1993 and 1997, comparing those started before the ‘90s reforms had taken hold with those completed between 1998 and 2001.

One interesting finding is that cost growth following procurement changes in the ’80s was greater for development than for production contracts. By comparison, the drop in cost growth following acquisition changes in the ’90s was greater for development contracts. Those results are especially noteworthy because the success of development contracts depends on trust between the government and the contractor, a trust POGO dislikes.

The two papers don’t tell the complete story. A scholar would want to know whether the results are sensitive to small changes in the selected years. However, taking into account the studies’ limitations, I think they show that data is better than anecdotes. Instead, people have let anecdotes dominate the debate about procurement changes.

Kelman is professor of public management at Harvard University’s Kennedy School of Government and former administrator of the Office of Federal Procurement Policy. He can be reached at steve_kelman@
harvard.edu.

FCW in Print

In the latest issue: Looking back on three decades of big stories in federal IT.

Featured

  • FCW @ 30 GPS

    FCW @ 30

    Since 1996, FCW has covered it all -- the major contracts, the disruptive technologies, the picayune scandals and the many, many people who make federal IT function. Here's a look back at six of the most significant stories.

  • Shutterstock image.

    A 'minibus' appropriations package could be in the cards

    A short-term funding bill is expected by Sept. 30 to keep the federal government operating through early December, but after that the options get more complicated.

  • Defense Secretary Ash Carter speaks at the TechCrunch Disrupt conference in San Francisco

    DOD launches new tech hub in Austin

    The DOD is opening a new Defense Innovation Unit Experimental office in Austin, Texas, while Congress debates legislation that could defund DIUx.

  • Shutterstock image.

    Merged IT modernization bill punts on funding

    A House panel approved a new IT modernization bill that appears poised to pass, but key funding questions are left for appropriators.

  • General Frost

    Army wants cyber capability everywhere

    The Army's cyber director said cyber, electronic warfare and information operations must be integrated into warfighters' doctrine and training.

  • Rising Star 2013

    Meet the 2016 Rising Stars

    FCW honors 30 early-career leaders in federal IT.

Reader comments

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group