Flaws in the evaluation process

The Department of Education's expert panel, assembled to find the best math programs in the United States, had a key problem, critics say. It relied heavily on studies of student achievement that were authored or co-authored by the directors of the programs themselves - or by people with close institutional or other ties to the program.

In the cases of Core-Plus Mathematics Project and the Connected Mathematics Project, a middle-school program, studies showing positive student achievement were submitted to the Department of Education's expert panel.

But neither Core-Plus nor Connected Math has yet published in a peer-reviewed journal the findings from the field-tests of their programs - though these were the primary studies supplied to the expert panel as proof the programs work.

"Peer review is essential," says Ronald Green, director of the Institute for the Study of Applied and Professional Ethics at Dartmouth College in Hanover, N.H. "To be independent, an evaluator's judgment must not be distorted by institutional relationships or personal ties or financial considerations."

Core-Plus was one of the best of the programs reviewed, panel members say. But studies of its effectiveness were co-authored by Harold Schoen, a University of Iowa professor.

Dr. Schoen, who is listed as a co-director of the program, admits he is in line to receive royalties from the sales of Core-Plus textbooks. His studies, he says, are not motivated by the prospect of royalties, of which he has received little.

But some critics have concerns. "You simply cannot have one of your principal investigators [in a research project] also be the outside evaluator," says R. James Milgram, a Stanford University mathematician and critic.

Nobody, including research ethicists, argue that Schoen's studies are invalid. As a co-director of the program, Schoen's studies provide a valuable basis for analysis. However, experts say, his co-authorship and receipt of royalties mean his reports should require independent peer review.

Still, Schoen and Steven Leinwand, the co-chair of the expert panel, both contend that the ultimate peer review for the winning programs was the education panel's process itself.

But outsiders say this was not a true peer review. For instance, the studies and programs were not anonymous to reviewers, thus opening the door to bias, says Thomas Loveless, senior analyst at the Brookings Institution in Washington.

Of 61 programs evaluated, 25 survived the quality-review process. Then, nine "impact reviewers," professional evaluators, examined supporting studies of student achievement for those programs.

Their summaries were then relied upon by the expert panel as evidence of a program's effectiveness. It was a last, critical step before voting.

Copies of the impact reviews for all 10 programs were obtained by the Monitor. These, along with interviews with the program directors, were used to determine which studies - and their authors - were used in evaluating achievement for Core-Plus and Connected Mathematics.

James Rutherford, a panel member, says reviewers "needed more manpower, time, and training to do a truly thorough and consistent job." Each program was supposed to be reviewed twice, but Connected Math and Number Power received only one impact review.

Requiring field-test results to be peer reviewed before being submitted would have helped lessen biases scientific research frequently has - especially when it is internally funded and has conflicting financial interests, Loveless and others say.

But Glenda Lappan, Connected Math's project director and former president of the NCTM, says her program is based on solid research, even if the first truly independent evaluations are just now rolling in.

"All the new [programs] are vulnerable to the criticism that the studies we all had conducted hadn't appeared in juried publications at that time," she says. "Many had been submitted for publication, but the process takes a long time."

The single impact review for Connected Math makes clear the importance of independent peer review.

The reviewer refers to "two strong studies" of student performance. The main one, "Effects of the Connected Mathematics Project on Student Attainment," is co-authored by Mark Hoover and two others. Yet during his stint as an evaluator, Mr. Hoover also worked and studied at Michigan State, the same university that received the Connected Math project grants of more than $7.7 million. This institutional affiliation violates common standards for independent evaluation, ethicists say.

"We're up front about the fact that the optics may not be real good on [Hoover]," says Dr. Lappan.

In the end, Hoover's study was the primary evidence for the panel. One other Connected Math study did figure in the single impact review. It was co-authored by David Ben-Chaim, listed then with the Weizmann Institute. But he produced his study with two others who were directors of the Connected Math team. Dr. Ben-Chaim's study was published in a peer-reviewed journal, but the Hoover study was the main one cited.

Some think these problems spring mainly from the difficulty of trying to find a silver bullet to quickly plug America's math gap.

"This is not a conspiracy," says Dr. Loveless. "What it shows is that the math-reform process is deeply flawed. The public has a right to expect government to be looking out for its interests in the most objective fashion the government can muster. It didn't do that."

(c) Copyright 2000. The Christian Science Publishing Society