Nature commentary: “Tighten the requirements for declaring physics breakthroughs”
DOI: 10.1063/PT.5.8125
Jan Conrad “wants to detect dark matter,” reports the Young Academy of Sweden , which reminds the public and researchers outside physics that this “would be a breakthrough ... and open up a whole new view of the Universe.” Conrad also has a deep interest in how physics breakthroughs should and shouldn’t be identified, as his 2 July Nature commentary explains, complete with bluntness about present publishing—and publicizing—practices and with prescriptions for improvement.
A Stockholm University webpage describes the work of Conrad’s experimental astroparticle physics group:
One approach is called indirect detection of particle dark matter with gamma rays using the satellite Fermi Large Area Telescope (LAT) as well as the Imaging Air Cherenkov Telescope H.E.S.S. We are also involved in preparations for the next generation Air Cherenkov Telescope Array CTA. Another way to look for particle dark matter is by detectors placed in deep underground mines, an approach dubbed direct detection. We are involved in the XENON programme as well as preparations for the next-to-next generation noble gas detector DARWIN .
In Nature, Conrad frames his argument by citing “a slew of announcements of major discoveries in particle astrophysics and cosmology” that have mostly “turned out to be false alarms,” which he sees as “the probable fate of the rest” as well. He lists “faster-than-light neutrinos; dark-matter particles producing γ-rays; X-rays scattering off nuclei underground; and even evidence in the cosmic microwave background for gravitational waves caused by the rapid inflation of the early Universe.”
He warns of consequences from “broadcasting seemingly extraordinary results to peers and the public before they are reviewed, or despite knowing that better data are just around the corner.” In science, the consequence is loss of credibility. In society, it’s undermined public trust in science.
He indicts trends that “have eroded traditional standards for making extraordinary claims": “intense competition, increased use of public data sets and online publishing of draft papers without proper refereeing.” And because of the “rash of false discoveries,” he condemns “three changes in the ways that scientific studies are done and reported.”
First, he charges that “statistical standards have fallen” and that “5-sigma claims are becoming rare as scientists rush to assert priority with exciting but tentative results.” He mentions disapprovingly two press releases: one that preceded the 2012 Higgs boson news at CERN and another about an 8 March arXiv preprint making a “dark-matter discovery claim.” He charges that the preprint’s authors acknowledged that a “software upgrade was imminent and would confirm or refute their claim, but did not wait for it.” He adds:
Detecting a noise fluctuation is nothing new, but the possibility that the “detection” might have been dark matter meant that it was widely reported in the media. Even balanced reporting raises the issue in the public’s mind; the account in the New York Times mentioned the non-detection, but the hint of excitement drove the story.
Conrad next argues against “the greater use of public data sets,” which “increases the risk that some researchers will make spurious detections near the edge of an instrument’s sensitivity.” He cautions that “analysis is difficult without inside information from those who built and calibrated the instrument.”
His third condemned change is that “many more papers are now released on preprint servers such as arXiv (which had about 100,000 submissions in 2014), and press releases are sent out before peer review.” He implicates “the rush to publish prematurely and publicize results.” He laments, “funding decisions are skewed; theorists waste a lot of time trying to devise explanations; and the public is misled through news reports.” He offers bluntly what he calls a “striking example of a premature claim released online before peer review,” namely, “the report last year of evidence for gravitational waves and cosmic inflation—the Universe’s rapid expansion in the instant after the Big Bang—by the BICEP2 microwave telescope at the South Pole":
The detection of a swirled polarization pattern, known as a B-mode, in the cosmic microwave background (radiation left over from the Big Bang) was not in doubt—it had a 7-sigma signal. But its supposed cosmic origin turned out to be false. It was shown six months later—with data from the European Space Agency’s Planck satellite—to be warm dust in the Milky Way. Again, I believe that the authors of the original paper must have known of the impending Planck data but chose to blow their trumpet ahead of confirmation. Eventually, the BICEP2 and Planck collaborations worked together to arrive at a solid result, an approach that should have been considered from the beginning.
But Conrad also prescribes measures for improvement. For “firm discoveries,” he calls for renewed respect for the 5-sigma rule “or an equivalent.” He counsels that scientific peers should vet researchers’ results before anything is publicly announced. “Quality assurance of preprints posted online should become stricter,” he declares. He continues:
An “endorsement system,” whereby users must be endorsed by other users before posting a paper, has been developed by arXiv to ensure that non-scientific pieces are not hosted there. More is needed for extraordinary claims. Named reviewers for major discoveries would reassure the readers and authors, as well as crediting the reviewers. Journals should discourage the referencing of arXiv papers.
Instrument builders and specialists who collected the original data should review major claims that are based on public data sets, either as referees, advisers or collaborators. Other teams with ancillary data that could refute or prove a claim should be involved in checking major results before release. This will require voluntary good conduct by competitors, which again could be encouraged by naming reviewers on breakthrough papers.
A system needs to be established to reward best practice. Collaborations should establish a way to ensure that a data team working with an individual scientist will not competitively sink the scientist’s publication nor diminish their visibility. Internal review should precede announcements of major results at conferences. Policies should be devised for author lists to give proper credit.
Journals and arXiv should find a strategy for allocating credit to the lead scientists in such collaborations. The BICEP2 team, for example, did work with the Planck collaboration later; had they been able to mark their priority better they might have delayed a press conference.
Conrad observes that the original BICEP2 paper illustrates how “incorrect papers” are often “more highly cited than counter cases.” He calls for devising academic metrics “that distinguish citations of discredited claims so that it is not more advantageous to state and retract a result than to make a solid discovery.”
He ends by calling for organizations like the American Physical Society and the International Union of Pure and Applied Physics to “lead a movement akin to the biology community’s reproducibility initiatives” and for scientists, publishers, and representatives of funding agencies to “convene to discuss improvements to norms such as peer review, metrics, use of databases, quality assurance and codes of conduct.”
---
Steven T. Corneliussen, a media analyst for the American Institute of Physics, monitors three national newspapers, the weeklies Nature and Science, and occasionally other publications. He has published op-eds in the Washington Post and other newspapers, has written for NASA’s history program, and is a science writer at a particle-accelerator laboratory.