the lectern banner

By Steve Kelman

Blog archive

OMB's evidence memo deserves praise

analytics concept art

By coincidence, only a few days before I wrote my most recent blog post on use of evidence to improve outcomes in emergency rooms after the Boston Marathon bombings and to reduce binge drinking at Dartmouth College, a memo came out from OMB Director Sylvia Mathews Burwell and three others in the White House about next steps in the evidence and innovation agenda. (Raise your hand if you didn't know there was an evidence agenda.)

Regardless of whether the "evidence agenda" in the White House is well-known, this memo is good news, in two respects. First, evidence is a good thing to help us make decisions. This statement is not necessarily as uncontroversial as it might sound. In political debates, Republicans have often been averse to evidence about phenomena in the natural world (climate change or evolution), while Democrats have often been averse to evidence about government programs, out of a worry that gathering evidence might show a lack of impact.

There is a second piece of good news in the memo as well. Traditionally, those promoting the use of evidence in government have often insisted somewhat dogmatically on what are often very expensive, long-duration, hard-to-execute randomized controlled trials to test the impact of government programs. (In such a trial, program results for people who are randomly assigned to receive a certain government intervention are compared with those who do not. In another form, people who planned to participate but didn't are compared with those who planned to participate and did.) 

The new memo, though it still seems to assign special status to such forms of evidence, also notes that companies are increasingly using "quick experiments" that in a fast and inexpensive way test much smaller interventions – the classic example would be different wordings on websites or in letters to people to see which produce a better response. These are particularly suitable for management interventions in workplace practices. They use the same principle of random assignment, but at less cost and time.

But econometric analyses, which use existing data rather than organizing new experiments, can also provide valuable evidence, though they are subject to more methodological worries than random experiments and hence their results are more subject to debate among researchers. Even the kinds of benchmarking studies used by organizations can be helpful, though often a smaller sample size and difficulty in drawing good conclusions about what makes one unit a good performer and another a bad one make conclusions from benchmarking more uncertain.

People who don't like the conclusions that evidence-based studies reach often prefer to shoot the messenger. Yes, there are poorly designed studies sponsored by advocates, intended to produce an outcome favorable to the product or idea being promoted.  Yes, there are often disagreements even among scholars about conclusions from studies, though these occur less often with experimental studies, whether random controlled trials or quick experiments. Yes, sometimes it is hard to gather data. And of course there are still values questions left in policy and management choices – is a program with a small impact for a fairly large cost worth it or not?

But surely using data to help inform these decisions is better than the alternatives, which dominate our political and governmental choices. So the OMB memo is to be welcomed.

 

Posted on Aug 02, 2013 at 1:47 PM


The Fed 100

Save the date for 28th annual Federal 100 Awards Gala.

Featured

  • computer network

    How Einstein changes the way government does business

    The Department of Commerce is revising its confidentiality agreement for statistical data survey respondents to reflect the fact that the Department of Homeland Security could see some of that data if it is captured by the Einstein system.

  • Defense Secretary Jim Mattis. Army photo by Monica King. Jan. 26, 2017.

    Mattis mulls consolidation in IT, cyber

    In a Feb. 17 memo, Defense Secretary Jim Mattis told senior leadership to establish teams to look for duplication across the armed services in business operations, including in IT and cybersecurity.

  • Image from Shutterstock.com

    DHS vague on rules for election aid, say states

    State election officials had more questions than answers after a Department of Homeland Security presentation on the designation of election systems as critical U.S. infrastructure.

  • Org Chart Stock Art - Shutterstock

    How the hiring freeze targets millennials

    The government desperately needs younger talent to replace an aging workforce, and experts say that a freeze on hiring doesn't help.

  • Shutterstock image: healthcare digital interface.

    VA moves ahead with homegrown scheduling IT

    The Department of Veterans Affairs will test an internally developed scheduling module at primary care sites nationwide to see if it's ready to service the entire agency.

  • Shutterstock images (honglouwawa & 0beron): Bitcoin image overlay replaced with a dollar sign on a hardware circuit.

    MGT Act poised for a comeback

    After missing in the last Congress, drafters of a bill to encourage cloud adoption are looking for a new plan.

Reader comments

Thu, Aug 8, 2013 Padrino

The fly in the 'evidence' ointment is that thsi view tends to drive policy in directions that are most easily measured. Example: It is easy to measure how kids retain facts; it is much harder to measure more complex intellectual tasks. The 'evidence' theme thus drives us to making those kids fact-retainers rather than thinkers. Many of the most important aspects of intelligent govt. policy involves things we don't know how to measure very well.

Tue, Aug 6, 2013 Steve Kelman

Thanks for this comment on pilots. This difference between quick experiments and pilots is that quick experiments have a control group (untreated or different treatment), while pilots usually don't. So you can compare two different kinds of message formats, or compare one group exposed to something and another group that isn't.

Tue, Aug 6, 2013

In the R&D world, this is called conducting a pilot.... where you quickly, cheaply, and on a small scale test an idea. If it succeeds, you adjust and conduct a larger demonstration. This applies for both airplanes, iPads, and Gmail.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group