the lectern banner

By Steve Kelman

Blog archive

What's so scary about performance ratings?

man graphs performance

A May 26 front-page article in The New York Times, with the scary title "Colleges Rattled as Obama Seeks Rating System," began by stating that college presidents are "appalled" by an Obama administration plan that "would compare schools on factors like how many of their students graduate, how much debt their students accumulate and how much money their students earn after graduating."

I am a professor, working at a university. I am also an advocate of using performance measurement to improve organizational performance. Should I feel "rattled" by the administration's effort?

The basic worries the college presidents expressed in the article were that the measurements could have perverse effects. (Sound familiar?)

Two in particular were cited in the article. The first was that measures of how much students on average earn after they graduate would penalize schools that trained more students in liberal arts, where salaries are lower compared to those for students with degrees in finance, and those that prepared students for poorly paid public-service careers such as social work or even teaching. A focus on post-graduation income would distort the behavior of universities towards a more-narrow view of their mission.

The second worry cited was that comparisons of debt load, or the measures of graduation rates and even post-graduation income, would discourage universities from taking a risk on students from disadvantaged backgrounds.

Do I share these worries?

I think the bottom line is that neither worry need argue against a performance rating system, if the system is designed correctly. A number of states, in looking at public school test scores, already perform what are called "value added" adjustments. The basic idea is that the scores are adjusted to account for factors that can produce differences in test scores that have nothing to do with the quality of the school. So, for example, schools where most of the kids come from poor families are going to have a harder time showing the same test scores as schools where the students' families are all rich. But the first school may actually be doing a better job –- adding more value -– than the second, and that's the difference we want to measure. Because these techniques often use a kind of statistical analysis that allows controlling for the impact of these irrelevant variables, I have often reminded my students, "This is why God invented multiple regression."

So in the case of the college ratings, if the scores are adjusted to control for percentage of liberal arts majors, emphasis on public service jobs and the economic backgrounds of the students, then the scores we see will come much closer to measuring their intended purpose –- which is to show differences in school teaching quality that influence graduation rates, students indebtedness and so forth. These performance measures can the play the role they are supposed to play for organizations, which is to spur performance improvements. (The administration has, in my view, over-emphasized the politically appealing punitive language about reducing federal funds to low performers, which puts the cart before the horse. Let's get a system working before we start playing around with financial penalties.)

There is, of course, a risk that the government will not introduce a value-added system, in which case the numbers might indeed create the problems about which presidents are worrying and warning. But I am sad that the universities seem to be looking for reasons this system might not work rather than embracing the performance improvement potential ratings can bring. Doctors have done the same thing over the years to try to prevent publication of hospital performance information. Unfortunately, both these efforts smell of special interest, and, although I'm at a university, I cannot endorse them.

Posted on May 27, 2014 at 8:16 AM


Reader comments

Fri, May 30, 2014 Al

I don't recommend Marvin's suggested scheme. One man's responsible citizen is another's annoying activist. Also, it invites a rejoinder: "Let me know when the performance measures include whether the graduate get divorces, raises their offspring, lives with their parents, succumbs to addiction or acquires a criminal record". These will just reflect what one individual thinks a responsible citizen is.

Fri, May 30, 2014 Al

The world penalizes students trained in the liberal arts following graduation. Why should the college be insulated from that hardship? If this information leads to a reduction in education prices in the liberal arts, how is this a bad outcome?

Wed, May 28, 2014

Colleges are their for students and this information should be made available to them prior to selection. If they have done their research, they will already know the starting salary of their chosen career. Those that are afraid of it, do have something to hide. Same with pay for performance in the fed world.

Wed, May 28, 2014 Marvin Sirbu

Robert Maynard Hutchins is famous for saying that "The object of the educational system, taken as a whole, is not to produce hands for industry or to teach the young how to make a living. It is to produce responsible citizens" Let me know when the performance measures include such factors as whether the graduate votes regularly, participates in town meetings and civic affairs, and remains abreast of the important political issues of the day.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above