the lectern banner

By Steve Kelman

Blog archive

OMB's evidence memo deserves praise

analytics concept art

By coincidence, only a few days before I wrote my most recent blog post on use of evidence to improve outcomes in emergency rooms after the Boston Marathon bombings and to reduce binge drinking at Dartmouth College, a memo came out from OMB Director Sylvia Mathews Burwell and three others in the White House about next steps in the evidence and innovation agenda. (Raise your hand if you didn't know there was an evidence agenda.)

Regardless of whether the "evidence agenda" in the White House is well-known, this memo is good news, in two respects. First, evidence is a good thing to help us make decisions. This statement is not necessarily as uncontroversial as it might sound. In political debates, Republicans have often been averse to evidence about phenomena in the natural world (climate change or evolution), while Democrats have often been averse to evidence about government programs, out of a worry that gathering evidence might show a lack of impact.

There is a second piece of good news in the memo as well. Traditionally, those promoting the use of evidence in government have often insisted somewhat dogmatically on what are often very expensive, long-duration, hard-to-execute randomized controlled trials to test the impact of government programs. (In such a trial, program results for people who are randomly assigned to receive a certain government intervention are compared with those who do not. In another form, people who planned to participate but didn't are compared with those who planned to participate and did.) 

The new memo, though it still seems to assign special status to such forms of evidence, also notes that companies are increasingly using "quick experiments" that in a fast and inexpensive way test much smaller interventions – the classic example would be different wordings on websites or in letters to people to see which produce a better response. These are particularly suitable for management interventions in workplace practices. They use the same principle of random assignment, but at less cost and time.

But econometric analyses, which use existing data rather than organizing new experiments, can also provide valuable evidence, though they are subject to more methodological worries than random experiments and hence their results are more subject to debate among researchers. Even the kinds of benchmarking studies used by organizations can be helpful, though often a smaller sample size and difficulty in drawing good conclusions about what makes one unit a good performer and another a bad one make conclusions from benchmarking more uncertain.

People who don't like the conclusions that evidence-based studies reach often prefer to shoot the messenger. Yes, there are poorly designed studies sponsored by advocates, intended to produce an outcome favorable to the product or idea being promoted.  Yes, there are often disagreements even among scholars about conclusions from studies, though these occur less often with experimental studies, whether random controlled trials or quick experiments. Yes, sometimes it is hard to gather data. And of course there are still values questions left in policy and management choices – is a program with a small impact for a fairly large cost worth it or not?

But surely using data to help inform these decisions is better than the alternatives, which dominate our political and governmental choices. So the OMB memo is to be welcomed.


Posted on Aug 02, 2013 at 1:47 PM

FCW in Print

In the latest issue: Looking back on three decades of big stories in federal IT.


  • Anne Rung -- Commerce Department Photo

    Exit interview with Anne Rung

    The government's departing top acquisition official said she leaves behind a solid foundation on which to build more effective and efficient federal IT.

  • Charles Phalen

    Administration appoints first head of NBIB

    The National Background Investigations Bureau announced the appointment of its first director as the agency prepares to take over processing government background checks.

  • Sen. James Lankford (R-Okla.)

    Senator: Rigid hiring process pushes millennials from federal work

    Sen. James Lankford (R-Okla.) said agencies are missing out on younger workers because of the government's rigidity, particularly its protracted hiring process.

  • FCW @ 30 GPS

    FCW @ 30

    Since 1987, FCW has covered it all -- the major contracts, the disruptive technologies, the picayune scandals and the many, many people who make federal IT function. Here's a look back at six of the most significant stories.

  • Shutterstock image.

    A 'minibus' appropriations package could be in the cards

    A short-term funding bill is expected by Sept. 30 to keep the federal government operating through early December, but after that the options get more complicated.

  • Defense Secretary Ash Carter speaks at the TechCrunch Disrupt conference in San Francisco

    DOD launches new tech hub in Austin

    The DOD is opening a new Defense Innovation Unit Experimental office in Austin, Texas, while Congress debates legislation that could defund DIUx.

Reader comments

Thu, Aug 8, 2013 Padrino

The fly in the 'evidence' ointment is that thsi view tends to drive policy in directions that are most easily measured. Example: It is easy to measure how kids retain facts; it is much harder to measure more complex intellectual tasks. The 'evidence' theme thus drives us to making those kids fact-retainers rather than thinkers. Many of the most important aspects of intelligent govt. policy involves things we don't know how to measure very well.

Tue, Aug 6, 2013 Steve Kelman

Thanks for this comment on pilots. This difference between quick experiments and pilots is that quick experiments have a control group (untreated or different treatment), while pilots usually don't. So you can compare two different kinds of message formats, or compare one group exposed to something and another group that isn't.

Tue, Aug 6, 2013

In the R&D world, this is called conducting a pilot.... where you quickly, cheaply, and on a small scale test an idea. If it succeeds, you adjust and conduct a larger demonstration. This applies for both airplanes, iPads, and Gmail.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group