the lectern banner

By Steve Kelman

Blog archive

What's so scary about performance ratings?

man graphs performance

A May 26 front-page article in The New York Times, with the scary title "Colleges Rattled as Obama Seeks Rating System," began by stating that college presidents are "appalled" by an Obama administration plan that "would compare schools on factors like how many of their students graduate, how much debt their students accumulate and how much money their students earn after graduating."

I am a professor, working at a university. I am also an advocate of using performance measurement to improve organizational performance. Should I feel "rattled" by the administration's effort?

The basic worries the college presidents expressed in the article were that the measurements could have perverse effects. (Sound familiar?)

Two in particular were cited in the article. The first was that measures of how much students on average earn after they graduate would penalize schools that trained more students in liberal arts, where salaries are lower compared to those for students with degrees in finance, and those that prepared students for poorly paid public-service careers such as social work or even teaching. A focus on post-graduation income would distort the behavior of universities towards a more-narrow view of their mission.

The second worry cited was that comparisons of debt load, or the measures of graduation rates and even post-graduation income, would discourage universities from taking a risk on students from disadvantaged backgrounds.

Do I share these worries?

I think the bottom line is that neither worry need argue against a performance rating system, if the system is designed correctly. A number of states, in looking at public school test scores, already perform what are called "value added" adjustments. The basic idea is that the scores are adjusted to account for factors that can produce differences in test scores that have nothing to do with the quality of the school. So, for example, schools where most of the kids come from poor families are going to have a harder time showing the same test scores as schools where the students' families are all rich. But the first school may actually be doing a better job –- adding more value -– than the second, and that's the difference we want to measure. Because these techniques often use a kind of statistical analysis that allows controlling for the impact of these irrelevant variables, I have often reminded my students, "This is why God invented multiple regression."

So in the case of the college ratings, if the scores are adjusted to control for percentage of liberal arts majors, emphasis on public service jobs and the economic backgrounds of the students, then the scores we see will come much closer to measuring their intended purpose –- which is to show differences in school teaching quality that influence graduation rates, students indebtedness and so forth. These performance measures can the play the role they are supposed to play for organizations, which is to spur performance improvements. (The administration has, in my view, over-emphasized the politically appealing punitive language about reducing federal funds to low performers, which puts the cart before the horse. Let's get a system working before we start playing around with financial penalties.)

There is, of course, a risk that the government will not introduce a value-added system, in which case the numbers might indeed create the problems about which presidents are worrying and warning. But I am sad that the universities seem to be looking for reasons this system might not work rather than embracing the performance improvement potential ratings can bring. Doctors have done the same thing over the years to try to prevent publication of hospital performance information. Unfortunately, both these efforts smell of special interest, and, although I'm at a university, I cannot endorse them.

Posted on May 27, 2014 at 8:16 AM

FCW in Print

In the latest issue: Looking back on three decades of big stories in federal IT.


  • Anne Rung -- Commerce Department Photo

    Exit interview with Anne Rung

    The government's departing top acquisition official said she leaves behind a solid foundation on which to build more effective and efficient federal IT.

  • Charles Phalen

    Administration appoints first head of NBIB

    The National Background Investigations Bureau announced the appointment of its first director as the agency prepares to take over processing government background checks.

  • Sen. James Lankford (R-Okla.)

    Senator: Rigid hiring process pushes millennials from federal work

    Sen. James Lankford (R-Okla.) said agencies are missing out on younger workers because of the government's rigidity, particularly its protracted hiring process.

  • FCW @ 30 GPS

    FCW @ 30

    Since 1987, FCW has covered it all -- the major contracts, the disruptive technologies, the picayune scandals and the many, many people who make federal IT function. Here's a look back at six of the most significant stories.

  • Shutterstock image.

    A 'minibus' appropriations package could be in the cards

    A short-term funding bill is expected by Sept. 30 to keep the federal government operating through early December, but after that the options get more complicated.

  • Defense Secretary Ash Carter speaks at the TechCrunch Disrupt conference in San Francisco

    DOD launches new tech hub in Austin

    The DOD is opening a new Defense Innovation Unit Experimental office in Austin, Texas, while Congress debates legislation that could defund DIUx.

Reader comments

Fri, May 30, 2014 Al

I don't recommend Marvin's suggested scheme. One man's responsible citizen is another's annoying activist. Also, it invites a rejoinder: "Let me know when the performance measures include whether the graduate get divorces, raises their offspring, lives with their parents, succumbs to addiction or acquires a criminal record". These will just reflect what one individual thinks a responsible citizen is.

Fri, May 30, 2014 Al

The world penalizes students trained in the liberal arts following graduation. Why should the college be insulated from that hardship? If this information leads to a reduction in education prices in the liberal arts, how is this a bad outcome?

Wed, May 28, 2014

Colleges are their for students and this information should be made available to them prior to selection. If they have done their research, they will already know the starting salary of their chosen career. Those that are afraid of it, do have something to hide. Same with pay for performance in the fed world.

Wed, May 28, 2014 Marvin Sirbu

Robert Maynard Hutchins is famous for saying that "The object of the educational system, taken as a whole, is not to produce hands for industry or to teach the young how to make a living. It is to produce responsible citizens" Let me know when the performance measures include such factors as whether the graduate votes regularly, participates in town meetings and civic affairs, and remains abreast of the important political issues of the day.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

More from 1105 Public Sector Media Group