An emerging challenge to science's credibility

A core tenet of science is the ability to reproduce an experiment's results. But worries are growing that many nonreproducible results are being accepted.

Concerns are mounting that a pillar of modern science is showing cracks.

A key feature of science is researchers' ability to reproduce experiments – to conduct a reality check on another group's work by using its materials and following its methods, then comparing the results.

It's a way to separate results worth building upon from those that aren't, either because a research team was careless, overlooked something, misinterpreted data, or at worst, fabricated results.

During the past several years, however, worries have grown that many nonreproducible results are working their way into the scientific literature, lingering undetected and, importantly, unchallenged. Such results can feed into others' work as they design their own experiments or pose their own research questions.

At stake, researchers say, is the credibility of science, especially when it is invoked to inform public policy on issues from climate change to new medical treatments. Investment decisions also hinge on credible, reliable research.

Concerns may grow further with the publication Thursday of what purports to be the most systematic effort to date to replicate others' experiments. More than 270 scientists around the world banded together to replicate 100 social- and cognitive-psychology experiments whose results appeared in three prestigious psychology journals.

While 97 percent of the original studies showed statistically significant results, only 36 percent of the replicated studies did. Of the successful replications, 83 percent showed a much smaller effects than the original studies showed, says Brian Nosek, a social psychologist at the University of Virginia and executive director of the Center for Open Science in Charlottesville, Va.   

The results come laced with caveats. Results are not necessarily wrong if they can't be replicated, nor are they necessarily right if they can be replicated, the researchers note. Numerous variables can come into play, yielding different results even when the same general procedures are followed. Moreover, the new study represents an initial look at a small sample of experiments in one discipline.

Indeed, it's unclear if these new results themselves could be reproduced, suggested Dr. Nosek, one of the study's coauthors, during a briefing this week on the work.

"This should just be seen as a first step, an initial piece of evidence for establishing what reproducibility in general might be," he said.

Still, he added, "the results suggest that there is a lot of room to improve reproducibility."   

Several factors can make a study challenging to replicate. Cutting-edge experiments can be expensive, its tools and procedures complicated, and it can take years to perform, notes Kelvin Droegemeier, vice president for research at the University of Oklahoma.   

"Due to that complexity, it's become more difficult to reproduce results," he says.

Moreover, competition is fierce for grant money, which typically doesn't reward redoing someone else's work. Instead it goes to work that looks to generate new results.

Researchers are under pressure to continually produce important results, which in turn keeps grant money coming in. A strong list of published results represents valuable currency for getting a faculty position at a university or tenure once on the faculty. These factors can combine in subtle and not-so-subtle ways to influence how some researchers design and execute studies and interpret their results.

For the new study, researchers worked closely with the original scientists to ensure that efforts to replicate the studies tracked the original procedures.

One member of the team, Wake Forest University social psychologist E. J. Masicampo, was replicating someone else's study even as another member tried to replicate one of his studies. 

His effort at replicating his assigned study found the original's effect as well. His own study fared less well, he said at the briefing.

It focused on the energy required to make difficult decisions when tired. He reasoned that giving someone a boost of energy when fatigued, in this case a sugary beverage, would provide the spark for a more thoughtful, "effortful" decision when asked to choose between an apartment close to or far from campus.

Why didn't his experiment replicate? A change in location, he says. His study took place among undergraduates at Florida State University. The attempt to replicate the results occurred at the University of Virginia, where the choice, he said, "was a no-brainer" to students.

In his case, one might argue no harm in a result that might be right in its initial context but not generalizable. That insight could lead research into more-productive directions – itself part of the scientific process.

But concerns about the persistence of unreproducible results lurking in the pages of science journals have prompted several efforts to deal with the issue. Among the examples:

• More than 500 research journals have signed on to a Center for Open Science effort to increase the availability of the data behind the studies they publish, as well as the computer code and methods used in the studies.
• Working with the National Institutes of Health (NIH), some 80 journals publishing biomedical studies agreed to guidelines that include a provision that raises the prospect that the journals would open their pages to studies that refute already-published work.
• The National Science Foundation, seeking to increase confidence in research it funds, has developed guidelines for transparent, reliable research that include looking for ways to encourage the publication of studies that replicate previous work or show negative results.

Efforts to boost research transparency and more strongly encourage researchers to take the time to replicate others' work can go a long way toward weeding out fraud, in addition to more quickly sorting out solid results from the dubious, researchers note.

Especially where research is high profile, an inability to reproduce results can be an effective corrective. For instance, last year the inability of other researchers to replicate results helped uncover research misconduct, including falsified data, in a potentially groundbreaking development in stem-cell research.

Like other careers, science is a human endeavor where to some extent incidents of fraud will always crop up, Oklahoma's Dr. Droegemeier says.

One way to deal with that is to make sure that when fraud is established, the hammer falls not only on the lead author of a the study, but on the paper's senior author as well, argues Mina Bissell, a cancer researcher and distinguished scientist at the Lawrence Berkeley National Laboratory in Berkeley, Calif.

The senior author typically is a professor or head of a lab, and it's their responsibility to ensure the integrity of the work done in their lab, she argues.

These young people come into the lab and "they don't know," she says. "They come into a big lab. Everybody's competing. The professor expects them to publish in Nature and Science. So they go an produce something that looks like Nature and Science."

And when someone challenges the results of a study based on an inability to replicate its results, it's critical for the two researchers or groups to work together to try to understand if and where the initial study – or the replication – ran off the rails, Dr. Bissell says.

Still, Droegemeier says, "you want to be very careful because you can really destroy someone's career even with an accusation" if at first blush a study doesn't jibe with efforts to replicate it.