'Mystery shopper' technique could be effective counterterrorism tool

The government should use proven tools to improve its counterterrorism performance

The business of connecting the dots of information received from diverse sources and then developing conclusions about terrorist threats is more complicated than often suggested in media accounts. Nonetheless, we have a right to demand that counterterrorism organizations measure their performance and learn how to refine their analysis methods so they do the best possible job of protecting us.

One approach that might help is for the National Counterterrorism Center, an organization established in 2004 specifically to work on connecting dots, to adopt a version of the “mystery shopper” technique that many businesses use to measure and improve their performance.

In traditional settings, retail companies hire a mystery shopping service to send “undercover” shoppers into their stores to collect information on the customer experience, identifying both strengths and weaknesses. Management then uses that data to devise training programs to improve customer service.

Here’s how it would work in the National Counterterrorism Center: A dedicated staff would be responsible for sending information about fictional individuals into different parts of the intelligence system, whether National Security Agency telephone intercepts or reports from agents on the ground. The unit would be responsible for tracking whether and how analysts evaluated the information — that is, did they successfully connect the dots?

The center’s performance would be measured by the percentage of cases the analysts correctly identified as exhibiting a high risk of terrorist behavior. The metric could be tracked by organization, by team and eventually — when enough cases have been assembled — by analyst.

Before starting, the mystery shopper unit would need to develop a consensus about which information patterns suggest a high risk of terrorist activities. And it would need to avoid planting too many high-risk rumors, which could overwhelm the center’s resources. So the unit would also need to send noise or low-risk information as well and track the percentage of good hits versus low-risk hits. A mystery shopper unit would also want to vary the types of information it sends to see if certain sources or types are less likely to be successfully identified.

Like performance measures in general, if such metrics are used as a management tool, they should improve the performance of the counterterrorism system. Establishing challenging performance targets will motivate employees to work harder and be more vigilant. Varying the kinds of information fed into the system will allow officials to diagnose its weaknesses so agencies can better target training or changes in techniques.

The center would need to create a learning environment in which honest errors are discussed but not punished — what professor Amy Edmondson, who has done research on how hospitals can best reduce medical errors, calls psychological safety.

The country needs the best possible performance from its National Counterterrorism Center. There are management tools that have proven effective for improving performance in many contexts outside counterterrorism. The government can no longer afford to ignore them.

About the Author

Kelman is professor of public management at Harvard University’s Kennedy School of Government and former administrator of the Office of Federal Procurement Policy. Connect with him on Twitter: @kelmansteve

The Fed 100

Read the profiles of all this year's winners.


  • Then-presidential candidate Donald Trump at a 2016 campaign event. Image: Shutterstock

    'Buy American' order puts procurement in the spotlight

    Some IT contractors are worried that the "buy American" executive order from President Trump could squeeze key innovators out of the market.

  • OMB chief Mick Mulvaney, shown here in as a member of Congress in 2013. (Photo credit Gage Skidmore/Flickr)

    White House taps old policies for new government makeover

    New guidance from OMB advises agencies to use shared services, GWACs and federal schedules for acquisition, and to leverage IT wherever possible in restructuring plans.

  • Shutterstock image (by Everett Historical): aerial of the Pentagon.

    What DOD's next CIO will have to deal with

    It could be months before the Defense Department has a new CIO, and he or she will face a host of organizational and operational challenges from Day One

  • USAF Gen. John Hyten

    General: Cyber Command needs new platform before NSA split

    U.S. Cyber Command should be elevated to a full combatant command as soon as possible, the head of Strategic Command told Congress, but it cannot be separated from the NSA until it has its own cyber platform.

  • Image from Shutterstock.

    DLA goes virtual

    The Defense Logistics Agency is in the midst of an ambitious campaign to eliminate its IT infrastructure and transition to using exclusively shared, hosted and virtual services.

  • Fed 100 logo

    The 2017 Federal 100

    The women and men who make up this year's Fed 100 are proof positive of what one person can make possibile in federal IT. Read on to learn more about each and every winner's accomplishments.

Reader comments

Tue, Feb 9, 2010 Kenny Burroughs

The analysis group needs more time to identify potential terrorists as the amount of data "explodes" that they are reviewing. Spending time finding fake terrorist will surely detract from spending time looking for the real ones. There has to be a better way than wasting security experts time playing what essentially is a game rather than looking for the next "real" bomber.

Tue, Feb 9, 2010 Mike Moxcey Fort Collins, CO

I think the idea is standard and foolish. Identifying high risk terrorist behavior has already been done and procedures set up. Adding in the mystery shopper is a _compliance_ measure now, to ensure that everybody in the analyst chain is following the identified procedures. It doesn't improve security; it merely measures the way we are already tracking data that we already believe is high risk. Typical security mentality thinks compliance with the rules means the rules work. We can use computers to force password complexity and length and prevent repeat use, but that essentially forces people to write their password down. Those security analysts subject to mystery shopper techniques will get very good at identifying what they are told to measure by those above them on the security chain. What they will miss is anomalous data and there will not be any incentive (or time) to investigate what will probably be the next style of terrorism to occur. As long as the initial identification of high risk behavior is correct, then measuring the compliance of the worker to meet that goal will help achieve the goal of better security. But if the terrorists change tactics due to security, then we are essentially tying the hands of analysts, forcing them to look for targets that have become irrelevant; fighting the last war so to speak instead of preparing for the next one.

Mon, Feb 8, 2010 Henry

This is great until the name used by the mystery shopper is close to yours and you end up on the no fly list and Local Law Enforcement is kicking your door in? How do you "un-connect" the dots, once you have entered false/msileading information into the system? There has got to be a better way.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group