To Find the Best Fit, Ignore the Rankings

As they do every fall, high school students and their parents are deciding on college lists -- figuring out where to apply and which colleges are on top of their wish lists. As is also the case every fall, U.S. News & World Report and others have released their rankings, suggesting which are the “best” colleges -- among all and in certain categories.

Many students gravitate (regardless of what the rankings say) to public institutions close to home. But many others rely on rankings to identify potential colleges -- or to convince parents that a particular institution is worth whatever it charges. And many colleges are banking on that, boasting about their scores in the latest rankings.

Educators have for years questioned the validity of the rankings, pointing out flaws with various parts of the methodologies of U.S. News and others.

A new study from researchers at Stanford University’s Graduate School of Education examines all of the evidence about rankings and comes to this conclusion: the best way to find a college that is a “good fit” is to ignore the rankings.

Notably, the finding isn’t based on abstract ideas about the value of education not being something that can be measured.

Rather, the analysis is based on research about factors many students (and parents) say they take into consideration when they evaluate potential colleges: student learning, well-being, job satisfaction and future income.

If you care about those factors, the rankings will not steer you well, the paper says.

In summarizing the paper’s findings, Denise Pope, a senior lecturer at Stanford’s education school, said, "Research tells us that the most successful students, both in college and beyond, are the ones who engage in the undergraduate experience regardless of how selective a school may be. This is almost always the case whether a student attends the top-ranked or 200th-ranked college."

Key factors in U.S. News and other rankings reward graduation rates and reputation. U.S. News has, over the years, placed more emphasis not just on raw graduation rates but “expected” graduation rates to reward institutions with higher than expected rates for students from at-risk populations.

But the Stanford study finds that graduation rates still reflect the student body being served more than the quality of the institution. And the study says there is no evidence linking reputation to anything but … reputation.

So reputation is “a self-fulfilling metric.”

Even measures that might seem inherently good -- such as the percentage of class sessions that are small -- may not actually mean much, the study says.

“While small classes are often seen as desirable, this metric is problematic … When schools offer more small classes, there is less space for students in those classes, meaning fewer students actually get to take them. Consider, as an extreme example, a school with 200 students where each student takes only one class. If there are nine classes with two students each, and one huge class with the remaining 182 students, the average class size at the school would still be only 20 students, and a full 90 percent of those classes would be considered ‘small’ classes.”

The report doesn’t say that it’s impossible to measure qualities about colleges that are important to students and parents. But real analysis would require examining the actual learning experience at colleges in ways rankings do not, the study says. For instance, the report cites Academically Adrift (University of Chicago Press), the 2011 book by Richard Arum and Josipa Roksa that found a lack of rigor and heft in most undergraduate syllabi, and a corresponding lack of learning. The findings were based on looking at what students were asked to read and write, among other things, in their courses. (Others have disputed Academically Adrift's findings)

“What does correlate with student learning?” the Stanford report asks. “Time spent studying. This is true regardless of institution and for all kinds of students. In other words, a student who studies hard at a nonselective school is very likely to learn, while a student who slacks off at a selective one is less likely to learn.”

Likewise the report points to research by Gallup on the factors in one’s college experience that predict whether one will appreciate that experience and find personally fulfilling work.

According to these Gallup surveys, the key predicting factor is not prestige of institution, but whether graduates look back on their college days and remember having had a professor who cared about them, made them excited to learn and encouraged them to follow their dreams -- which Gallup called being "emotionally supported" while in college. (Note: Inside Higher Ed works with Gallup on some surveys but did not play a role in this research.)

Broadly, Gallup identifies a series of practices for students to seek out at any college as likely to increase their engagement with learning -- and their satisfaction. These include “taking a course with a professor who makes learning exciting,” and “working with professors who care about students personally” and working on projects across several semesters.

The Stanford report generally suggests that students looking at potential colleges focus on finding places where the right conditions exist for the students to make the most of their opportunities. And that kind of review needs to focus on concrete measures or conditions that typically aren’t what makes a college rise or fall in U.S. News.

A key point of the report is that there is no single right focus for a college search -- and that the selectivity-driven measures used in most rankings may be appropriate for a minority of students, but misdirect most others.

“For some students, deciding where to go to college may depend on academic factors, such as access to cutting-edge researchers in a beloved field with opportunities to be involved in graduate-level work, or a well-established professional preparation program,” the report says. “For others, the decision might be influenced by location: a college close to home, or far away, in a small town or in a big city. Or it might have to do with extracurricular programming: a robust student activities program, intramural sports, or the arts.

"The decision might include cultural opportunities: a university with a strong international languages program, a culturally diverse student body, and a track record of successful study-abroad exchanges. The presence or absence of a Greek system or a large Division 1 athletics program might be important factors for some students. Perhaps religious denomination, institutional size, or comprehensive support for those with learning differences may sway the decision. And of course, financial aid and cost are key factors as well.”

The report adds that “rather than choosing a school based primarily on a flawed scoring system, students should ask whether they will be engaged at the college in ways that will allow them to form strong relationships with professors and mentors, apply their learning via internships and long-term projects, and find a sense of community.”

Robert Morse, who leads the college rankings effort at U.S. News, said via email that he and his colleagues were still reviewing the Stanford analysis.

He noted that the paper did not reflect the most recent changes made by U.S. News in its methodology, changes that the magazine says placed a greater emphasis than in the past on rewarding colleges that promote social mobility. (Critics have said that the change are largely cosmetic and that the rankings continue to reward colleges primarily for being wealthy and serving the best prepared and generally wealthiest students.)

Morse also noted that U.S. News does not just release annual rankings, but lists of top academic programs, colleges with good internship programs, innovative study abroad and more.

Just because U.S. News publishes rankings, Morse said, doesn’t mean it believes that’s the only way students should evaluate colleges. “We have always encouraged consumers to use the rankings as a start,” he said.