Best in which show?

A new survey challenges how the heavy hitters in college rankings decide which schools are on top

Two years ago, Kim Frankwick was hunting for the college of her dreams. But unlike her friends, she decided not to use the popular US News & World Report "America's Best Colleges" rankings to help her choose.

Instead, she used a small guidebook called "Colleges That Change Lives" - and ended up at Beloit College in Wisconsin. Today, the sophomore Russian major is happy - especially that she didn't rely on rankings.

"The quality of the teaching and learning here is incredible," she says. "I don't know if I would have found Beloit through those rankings."

She is not alone in wondering about such things.

Each fall, as some college presidents preen over their school's hot new ranking, many others grouse that rankings based on measures such as the size of the campus library fail to answer the question at the heart of choosing any college: What are students on campus actually learning?

That concern has not slowed what has become a rankings juggernaut. Money Magazine, the Princeton Review, Yahoo! (the nation's "most wired" institutions), Time, Newsweek, Kiplinger, and Mother Jones have all jumped into the rankings game in the past decade. Along with US News, they funnel such factors as retention rates, faculty-student ratios, and graduation rates into their own formulas to calibrate "the best."

But interest is growing in alternative ways to measure an institution's strong points. Some observers worry the "best" may really just be the biggest or richest or oldest. Lost in the rankings snapshot may be important information about the educational quality of students' experience.

Indeed, some institutions are becoming more vocal in their opposition to the traditional rankings approach. Certain colleges, like St. John's College in Annapolis, Md., don't report to US News anymore - and say applications are up anyway.

A new approach

Now arrives a new entrant that's not a ranking at all - but could one day turn the rankings game on its ear by gauging internal institutional quality based on student learning, not external data points.

The new National Survey of Student Engagement, released yesterday, queried 63,000 students at 276 universities and colleges last spring. The focus: how much time they spent in activities that research has shown cause students to learn.

For instance, it asked seniors and freshmen on each campus how much time they spent discussing, analyzing, synthesizing. It tracked how often they collaborated with friends, and how many books they read on their own and how many they were assigned. It sought information on how many reports of 20 pages or more students had written. It asked if they had given a class presentation, and if so, how frequently (see chart, left).

"This is unveiling a dimension of educational quality not revealed by the usual rankings," says Russell Edgerton, director of the Pew Forum on Undergraduate Learning, a sponsor with the Carnegie Foundation for the Advancement of Teaching. "A lot of people think quality is measured by selectivity in admissions," he says. "This is a different picture."

Indeed, George Kuh, director of the NSSE at Indiana University, Bloomington, calls it a chance to peer inside "the black box." "All colleges say they are in the business of learning," he says. "In many cases evaluating quality is a matter of measuring time devoted to task. The more [students] write, the better they become at expressing themselves in the written word."

They are quick to acknowledge that because this is only the first year of the survey, it doesn't yet include many top-ranked four-year institutions. But a number of prestigious schools, such as Rice University, the University of Virginia, and George Mason University, participated. NSSE hopes to greatly expand the list each year.

Much research has examined learning in classrooms and disciplines. But the NSSE measures a broader swath, looking across the surveyed institutions in five areas: academic challenge, active and collaborative learning, student interaction with faculty, enriching experiences, and supportive environment.

"This allows us to get inside what actually goes on in the central activity of the college in teaching and learning," Dr. Kuh says. "This is information that has not been publicly reported before."

What students had to say

Among the surprises extracted from the 276 institutions, from doctoral universities to liberal-arts colleges:

* Almost a fifth (19 percent) of all first-year students "never" made a class presentation.

* Most first-year and senior students (about 90 percent) say they at least "occasionally" work harder than they thought they could to meet instructors' standards.

* Almost half (49 percent) of first-year students and three-fifths (62 percent) of seniors "often" or "very often" got prompt feedback from teachers.

* About four-fifths (79 percent) of all students say their institutions emphasize spending significant time on studying. Yet less than 15 percent of full-time freshmen and seniors spent 26 or more hours a week outside of class studying. About half (47 percent) spent 6 to 15 hours a week, far less than the recommended two hours of study for each class hour.

Such contradictory findings could help institutions take action to remedy the "mismatch" between what they say they want in terms of studying and the level of performance to which they actually hold students accountable, the report says.

"These students [who study little] are not flunking," Kuh says. "If students can get by with this little effort, can we be satisfied with the quality of the educational experience? This is kind of staggering."

Who's at the top?

The NSSE also revealed another point of interest: three of four best-performing institutions in the survey were small, little-known liberal-arts colleges with fewer resources than large doctoral institutions.

A reader of US News & World Report rankings of "America's Best Colleges" would eventually find Beloit College, Elon College, Sweet Briar College, and Centre College. Centre and Beloit were ranked 42nd and 47th, respectively, among the top 50 US liberal-arts colleges. Sweet Briar was in the second tier. Elon ranked 11th among "Southern Universities."

Yet these four stood out on the NSSE - getting attention they might not otherwise receive - because they were the only schools scoring in the top 20 percent of all five of NSSE's key categories.

"It's nice to be on the list of top schools, but it's even better to have it based on serious research and not just reputation," says William Flanagan, Beloit's dean of students. "It affirms that what we think is important is being felt by students."

Little of the information parents and students use in their college research reveals factors like the amount of writing on campus, he says, or the real level of connection with faculty, or contact between students of different cultures.

But while the NSSE survey does those things, it reports them by institutional group only. NSSE had to promise not to release individual school results unless permitted.

Asking for results

Still, parents and students can find out which schools were surveyed in the report (www.indiana.edu/~nsse/) and see if their prime choices are included - then ask those specific schools for results.

Predictably, perhaps, some participants would like to see NSSE results made public in some sort of ranking. "I would be thrilled if somebody took a stab at saying, this is what students say about going there," says John Roush, president of Centre College in Danville, Ky. "There are a lot of state and other institutions claiming they're individualized and intimate, with classes of 200 to 400 students - the truth is they're not."

Legislators pushing for accountability may demand survey results from state schools. Experts also say the NSSE is just in the vanguard of a new wave of reports exploring college quality. Later this month, a new study will offer a 50-state report card using many measures not currently in most rankings.

"There's relentless pressure for this information," says Patrick Callan, director of the National Center for Public Policy and Higher Education, which is producing the report card. "It's becoming critical - whether you're a student at home with parents trying to decide where to attend, or a legislator trying to decide where the next million bucks goes."

E-mail claytonm@csps.com

(c) Copyright 2000. The Christian Science Publishing Society