Bad performance metrics revisited

Steve Kelman explores what it takes to make sure managers are measuring the right things.

Shutterstock image (by Ismagilov): charting data on the wall.

In January, just as I was returning to teaching after a year's absence due to my illness, I wrote a blog post about a conversation I'd had in an executive education class at the end of three sessions on using performance measurement. I had asked the participants to think of a performance measure currently being used in their organization that they thought created problems. Then I asked them whether they would favor simply getting rid of it, or if not, what they would do to improve it. When I called on a few people during our class that day, those who spoke cited metrics they would like to nix -- but couldn't because they had been imposed from the outside.

This year, I asked the same questions, but improved my data collection. I gave participants three alternatives and had everyone vote in class using clickers, so I could get results for the whole cohort. The three alternatives were:

  1. scrap metric if I can
  2. keep measure as is (can't come up with better idea, and better than nothing), and
  3. suggest a change in the metric

The percentages revealed in the clicker poll were fascinating. Only 11 percent said they would simply scrap the metric, while 19 percent said they would continue to accept it without change despite its imperfections.

This latter reaction reflected an argument I call "compared to what?" which is often ignored in discussions of performance measurement. It is tempting to say that if our measures are imperfect and that performance using such imperfect measures is worse than using perfect ones, we shouldn't manage using such measures. But that is mistaken, because the right comparison is performance using no measures at all. If performance using imperfect measures is better than using no measures at all, those imperfect metrics may well be worth keeping.

The most interesting number was the 70 percent stating they would work to improve the measure; that is good news about prospects for the success of performance measurement in government.

Anyone who reflects on performance measures, whether used in government or elsewhere, knows they are typically imperfect and create some level of dysfunctional effects, with people seeking to meet even a problematic measure. The question is whether, given that fact, we should just wring our hands and bemoan our fate -- or whether we should try to take control of our destiny and, as intelligent human beings, try to improve the imperfect measures over time.

I often cite the example of Army physical fitness metrics that became criticized for not reflecting physical challenges on actual battlefields: the Army got a committee together to develop better metrics that were more realistic. Some version of this seemed to be what my students were doing in their organizations.

Here were the specific examples cited in class:

  • Managers in an intelligence organization were using measures of how much schooling applicants had, and their background in relevant area studies (e.g. Middle East). They were unhappy with the quality and the performance of applicants they got using these screens. So hiring managers put their heads together to see what they thought made for a successful analyst. It was creativity and general analytic ability, they concluded. They then, with the help of a psychologist, changed their screening to select for those abilities. The quality of new hires under the revised metrics went up.
  • The Office of Management and Budget had given one program within an agency a goal for fielding certain discoveries from an ambitious new research effort that the program felt went beyond what it could do by itself. After conversations with OMB, the goal was changed to one shared by several programs, as part of an ecosystem, and its attainment divided into several chunks.
  • A metric for timeliness in closing out contracts did not reflect what was under the closeout agency's control because it included contracts that could not be closed out pending a final audit report from the Defense Contract Audit Agency. The agency changed the metric to measure closeout timeliness for contracts available to be closed out.
  • A metric for speed in issuing an agency's regulations did not distinguish between simple and complex regulations. The agency introduced a classification scheme for regulation complexity, and began to measure performance separately for the two categories.
  • One agency's metric stipulated a 45-day turnaround time for investigations. In this case, after discussions the agency decided not to change the metric itself, but to recognize that managers would have some wiggle room if they could explain to superiors situations where a suddenly emerging priority made reaching that goal impossible for a while.

These examples were interesting, and all reflected a refusal to simply be a victim of one's fate. However, the discussion also suggested a problem with agencies' use of performance measures. None of those who discussed problems with a turnaround-time metric -- and, notably, three of the five examples brought up in class involved processing speed -- suggested using a metric of the quality of reviews to complement one for speed. And if the only thing you measure is speed, it creates an incentive for reviews that are quick but superficial.

Indeed, earlier in the very same class, when students were talking about an effort in the D.C. city government to improve throughput time for face-to-face transactions at the Department of Motor Vehicles, two students actually had noted that this metric created an incentive for sloppy handling of customers. I had suggested earlier in class that when one performance measure leaves out an important dimension of performance, the classic solution is to develop an additional measure to represent that dimension.

I am guessing that for many agencies that use some measure of processing time, this processing is an important part of what they do, and that therefore it would be worth putting resources into measuring quality -- even if only on a sample basis. (Indeed, one could imagine a quality metric used instead of a speed metric, though this would create the opposite incentive to process at molasses speed.) Yet the apparent lack of using both quality and speed metrics to complement each other suggests an area where agencies still need to make progress in thinking about how best to do performance measurement to improve government.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.