Research

Study Questions Popular Measures of Student Opportunity in College Rankings

• By Dian Schaffhauser
• 02/11/19

Often, national rankings of universities attempt to highlight those schools providing the greatest opportunities for low-income students. The problem is that they're, well, national: They don't take into account regional factors that may paint a very different picture. A new measure proposed by a couple of economics professors aims to remedy this by allowing universities to judge their recruitment efforts against their own missions and circumstances.

The alternative, developed by Caroline Hoxby, a professor of economics at Stanford University and a senior fellow of the Hoover Institution, and Sarah Turner, a professor of economics and education at the University of Virginia, is laid out in a paper published by Education Next. Education Next is a scholarly journal "committed to careful examination of evidence relating to school reform." The research was drawn from "Measuring Opportunity in U.S. Education," initially published as a working paper by the National Bureau of Economic Research.

According to the authors, recent efforts by federal lawmakers to develop rankings of colleges and universities by the percentage of low-income students they enroll requires the use of "popular measures of 'opportunity.'" These measures basically rely on data put through one of three formulas — the share of students receiving federal Pell grants (the "Pell Share"), the share of students from the bottom 20 percent of the national income distribution ("Bottom Quintile") and a calculation of "Intergenerational Mobility" — all of which ultimately do comparisons of enrollment against national averages. As the paper asserted, those approaches fail to take into account the differences between institutions' available pools of potential students. As a result, institutions are mistakenly penalized or rewarded.

As an example, the authors explained, if the University of Maine and the University of Connecticut served every student in their state who met their academic standards, Maine would draw 22 percent of students from families with incomes below $40,000 compared to 10 percent at the University of Connecticut. The University of Connecticut would face penalties because it faced an income distribution with a high average. The University of Maine would be rewarded because average incomes in its relevant pool were low.

The Intergenerational Mobility measure, on the other hand — which reflects the share of an institution's students who are from the national bottom 20 percent and who end as adults in the top 20 percent — punishes universities "twice over" for working with a relevant pool with high income equality and rewards those working with a pool that has high income inequality. Here's how that works. The University of Wisconsin has unusual income equality, which means its students are unlikely to come from that national bottom quintile and also unlikely to end up in that top national quintile. However, the University of California, which works with a disproportionate number of ultra-poor and ultra-rich students, would be "slated for rewards" because of the existing inequality among its student pool.

Universities that are "disproportionately successful at enrolling low-income students end up with low rankings on the popular measures," the researchers noted. Likewise, those that are "disproportionately unsuccessful" can land at the top of the rankings. For instance, in a comparison of all 50 flagship state universities, those in Illinois, Connecticut and Wisconsin ranked near the bottom on the Pell Share and Bottom Quintile measures. It was just the reverse for the universities in Maine, Montana and New Mexico, all of which looked stellar on the popular measures. Yet, under a new "pool-based" measure proposed in the study, the results would be just the opposite.

Rather than comparing an institution's student enrollment to national averages or only focusing on low-income students, a relevant-pool measure would allow these schools to: a) "see the big picture of enrollment"; and b) focus on a particular part of the income distribution that interested them. The pool would consist of a profile of students based on geographic location and academic mission.

The working paper at the heart of the upcoming report encouraged all institutions to undertake a "rigorous-data-driven self-examination" to define their relevant pools, compare their students to the pool and draw "lessons for recruitment, admissions and financial aid." By doing so, those low-income, high-achieving students who could benefit by attending selective institutions would benefit, the authors suggested, even as the universities themselves furthered "both equity and excellence simultaneously."

The research provided examples to illustrate the advantages of the new method being proposed. As the authors explained, their approach would show how each university is enrolling students across the whole of its relevant pool's income distribution; would offer a measuring stick without imposing a given mission on the school; and would let the institution choose its own targets across the income distribution, which might include enrolling low-income students disproportionately. The method also wouldn't "encourage perverse behavior such as neglecting students just above an arbitrary income threshold," they noted. By eliminating that, institutions are less likely to game the system by pouring their recruitment efforts into "threshold-eligible students ... at the expense of other modest-income students."

"We have purposely avoided ranking all institutions because that would require us to assert the relevant pool — and thus the mission — of thousands of institutions of higher education in the United States. This is not our right," wrote Hoxby and Turner. "A university that used our proposed relevant-pool-based measure would not find a conflict between pursuing its mission and providing opportunities to students regardless of background."

The initial working paper is available on a Stanford website. An article summarizing the new measure is available on the Education Next website.