the lectern banner

By Steve Kelman

Blog archive

OMB's evidence memo deserves praise

analytics concept art

By coincidence, only a few days before I wrote my most recent blog post on use of evidence to improve outcomes in emergency rooms after the Boston Marathon bombings and to reduce binge drinking at Dartmouth College, a memo came out from OMB Director Sylvia Mathews Burwell and three others in the White House about next steps in the evidence and innovation agenda. (Raise your hand if you didn't know there was an evidence agenda.)

Regardless of whether the "evidence agenda" in the White House is well-known, this memo is good news, in two respects. First, evidence is a good thing to help us make decisions. This statement is not necessarily as uncontroversial as it might sound. In political debates, Republicans have often been averse to evidence about phenomena in the natural world (climate change or evolution), while Democrats have often been averse to evidence about government programs, out of a worry that gathering evidence might show a lack of impact.

There is a second piece of good news in the memo as well. Traditionally, those promoting the use of evidence in government have often insisted somewhat dogmatically on what are often very expensive, long-duration, hard-to-execute randomized controlled trials to test the impact of government programs. (In such a trial, program results for people who are randomly assigned to receive a certain government intervention are compared with those who do not. In another form, people who planned to participate but didn't are compared with those who planned to participate and did.) 

The new memo, though it still seems to assign special status to such forms of evidence, also notes that companies are increasingly using "quick experiments" that in a fast and inexpensive way test much smaller interventions – the classic example would be different wordings on websites or in letters to people to see which produce a better response. These are particularly suitable for management interventions in workplace practices. They use the same principle of random assignment, but at less cost and time.

But econometric analyses, which use existing data rather than organizing new experiments, can also provide valuable evidence, though they are subject to more methodological worries than random experiments and hence their results are more subject to debate among researchers. Even the kinds of benchmarking studies used by organizations can be helpful, though often a smaller sample size and difficulty in drawing good conclusions about what makes one unit a good performer and another a bad one make conclusions from benchmarking more uncertain.

People who don't like the conclusions that evidence-based studies reach often prefer to shoot the messenger. Yes, there are poorly designed studies sponsored by advocates, intended to produce an outcome favorable to the product or idea being promoted.  Yes, there are often disagreements even among scholars about conclusions from studies, though these occur less often with experimental studies, whether random controlled trials or quick experiments. Yes, sometimes it is hard to gather data. And of course there are still values questions left in policy and management choices – is a program with a small impact for a fairly large cost worth it or not?

But surely using data to help inform these decisions is better than the alternatives, which dominate our political and governmental choices. So the OMB memo is to be welcomed.

 

Posted on Aug 02, 2013 at 1:47 PM


Featured

Reader comments

Thu, Aug 8, 2013 Padrino

The fly in the 'evidence' ointment is that thsi view tends to drive policy in directions that are most easily measured. Example: It is easy to measure how kids retain facts; it is much harder to measure more complex intellectual tasks. The 'evidence' theme thus drives us to making those kids fact-retainers rather than thinkers. Many of the most important aspects of intelligent govt. policy involves things we don't know how to measure very well.

Tue, Aug 6, 2013 Steve Kelman

Thanks for this comment on pilots. This difference between quick experiments and pilots is that quick experiments have a control group (untreated or different treatment), while pilots usually don't. So you can compare two different kinds of message formats, or compare one group exposed to something and another group that isn't.

Tue, Aug 6, 2013

In the R&D world, this is called conducting a pilot.... where you quickly, cheaply, and on a small scale test an idea. If it succeeds, you adjust and conduct a larger demonstration. This applies for both airplanes, iPads, and Gmail.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above