New York Times columnist seeks scientists’ views on irreproducibility
DOI: 10.1063/PT.5.8036
John P. A. Ioannidis ‘s 2005 PLOS Medicine paper “Why most published research findings are false” continues to resound. On 31 January, for example, the President’s Council of Advisors on Science and Technology examined the issue, often called irreproducibility. Nature maintains a special archive on it.
Now New York Times science columnist George Johnson is asking scientists to comment. He observes that “physical science—everything from geology and climatology to cosmology and particle physics” has not “received the same kind of scrutiny” as other fields. He wonders whether irreproducibility is less common in physical science.
The introduction to Nature‘s archive begins without excluding physical science: “No research paper can ever be considered to be the final word, and the replication and corroboration of research results is key to the scientific process.” But the second sentence narrows the focus to the life sciences: “In studying complex entities, especially animals and human beings, the complexity of the system and of the techniques can all too easily lead to results that seem robust in the lab, and valid to editors and referees of journals, but which do not stand the test of further studies.”
Nature‘s 24 April 2013 editorial “Announcement: Reducing our irreproducibility” employed that narrowed focus in promising, “From next month, Nature and the Nature research journals will introduce editorial measures to address the problem by improving the consistency and quality of reporting in life-sciences articles.” The promised measures included increased attention to experimental methods, provision of unlimited space for descriptions of methods and of experimental design, encouragement of the posting of data, and additional rigor in assessing statistical practices.
So what about the physical sciences in all of this? Johnson discussed that with Ioannidis, who holds several professorial titles at the Stanford University School of Medicine.
“Physical sciences have a stronger tradition of some solid practices that improve reproducibility,” [Ioannidis said] in an email. Collaborative research, for example, is customary in physics, including large consortiums of experimenters like the teams that announced the discovery of the Higgs particle. “This certainly increases the transparency, reliability and cross-checking of proposed research findings,” he wrote.
He also mentioned more stringent statistical standards in particle physics—like the five sigma measure I mentioned in my second column —as well as sociological factors: “There seems to be a higher community standard for ‘shaming’ reputations if people step out and make claims that are subsequently refuted.” Cold fusion was a notorious example. He also saw less of an aversion to publishing negative experimental results—that is, failed replications.
Another factor...is how constrained a field is in generating plausible hypotheses to test. Almost anything might be suspected of causing cancer, but physicists are unlikely to propose conjectures that violate quantum mechanics or general relativity.
Johnson ended the column by asking scientists to comment:
Dr. Ioannidis said he was struck by an “arrogant dismissal” by some physical scientists of the suggestion that their field might be anything less than pristine. We won’t really know, he said, unless there are empirical studies like the recent ones in medical science.
“I have no doubt that false positives occur in all of these fields,” he concluded, “and occasionally they may be a major problem.”
I’ll be looking further into this matter for a future column and would welcome comments from scientists about the situation in their own domain.
Johnson can be contacted via his page at the Times.
---
Steven T. Corneliussen, a media analyst for the American Institute of Physics, monitors three national newspapers, the weeklies Nature and Science, and occasionally other publications. He has published op-eds in the Washington Post and other newspapers, has written for NASA’s history program, and is a science writer at a particle-accelerator laboratory.