Why aren't drug tests reproducible?
In the “Annals of Science,” Jonah Lehrer asks “is there something wrong with the scientific method?” He poses the question in an article entitled “The Truth Wears Off” in the December 13, 2010 issue of The New Yorker (pp. 52-57). The problem is that across disciplines “claims that have been enshrined in textbooks are suddenly unprovable.”
It is a problem of being unable to reproduce results in subsequent experiments. Even scientists who perform the original experiment cannot reproduce their own results. The pattern is that, over time, results become less strong or even disappear. Again, it is occurring in many disciplines but is especially acute in medicine. For instance, the original tests showed great promise for second-generation antipsychotic drugs and they became the most profitable products for some drug companies. New tests show the second-generation drugs no more effective than first-generation antipsychotics in use since the 1950s. In some cases, the new drugs perform worse.
The same thing is happening with cardiac stents, vitamin E therapy, and antidepressants. The decline in the efficacy of antidepressants is especially dramatic.
There are similar case studies detailed in psychology and zoology. There is a widespread problem of the non-reproduciblity of experimental results. The problem is well known, but scientists understandably don’t want to talk about it publicly. One scientist was advised not to attempt to reproduce his own results as he would only be disappointed. Results long since called into questions remain as truths in textbooks. Medical treatments undermined by subsequent tests remain in widespread use.
The author discusses all the possible reasons for this phenomenon from poorly structured experiments, to non-randomness and professional bias. All are likely sources of experimental failure, but he enumerates attempts by serious scientists to overcome all such weaknesses to no avail. For whatever reason, it is becoming increasingly difficult to reproduce experimental results.
The author is understated and balanced in his presentation. He offers no definitive explanation for what he reports. He ends on an almost Weberian note.
Add/view comments on this post.
------------------------------
The Christian Science Monitor has assembled a diverse group of the best economy-related bloggers out there. Our guest bloggers are not employed or directed by the Monitor and the views expressed are the bloggers' own, as is responsibility for the content of their blogs. To contact us about a blogger, click here. To add or view a comment on a guest blog, please go to the blogger's own site by clicking on the link above.
