New tool measures thoroughness of peer review reports

The level of detail of peer review reports is consistent across physical science journals regardless of the publications’ prominence, according to a first-of-its-kind large-scale analysis of reviewer reports in different scientific fields. Previous studies have evaluated the length of peer reviews and the demographics of the referees , but it has been difficult to examine the content of reviews because journals rarely disclose them.

For their June study in PeerJ , Flaminio Squazzoni, a sociologist at the University of Milan, Italy, and his colleagues got access to 1.3 million referee reports submitted between 2018 and 2020 to 740 scholarly journals issued by Dutch publishing giant Elsevier, including 161 physical science titles such as Physics Letters B and the Journal of Computational Physics.

The study authors created an algorithm that uses techniques from natural-language processing to sift through reviewer reports and determine whether they mention 10 different elements, including comments on relevant previous literature, the applicability of the study, and the impact and limitations of the study’s findings. The criteria come from a 2020 proposal from another team for measuring the quality of peer review in biomedical literature. The scores for each of the 10 criteria were given equal weight in the calculation of an overall measure that the authors called the developmental score.

Social sciences and economics journals generally had the highest developmental scores of the fields studied, which included the life sciences and health and medical sciences in addition to the physical sciences.

The researchers found that the reviews in the physical sciences had similar developmental scores regardless of the impact factor of the journal. That contrasts with the social sciences and life sciences, where review scores tended to correlate directly with journal impact factor. Squazzoni thinks that’s because reviewers in the physical sciences have a more standardized approach than do scientists in other fields.

Compared with those in other fields, physical science reviewers are more concerned about how results are presented and less about the impact of the findings. The mean developmental scores in the physical sciences are highest for methods—assessment of materials, methods, and study design—and lowest for references to data availability.

Overall, junior academics generally produced reports with higher developmental scores than those of senior faculty, the study found. “Their reports are more constructive, more informative, longer, and more detailed,” Squazzoni says. He says that may be because senior faculty have less time to spend on peer review, or because junior researchers want to show how diligent and reliable they are so that journal editors consider them for future submissions. Reviews by women tended to have higher developmental scores than those by men, the study found, and reviewers based in Western Europe tended to have higher scores than those working in other parts of the world.

Ludo Waltman, professor of quantitative science studies and deputy director of the Center for Science and Technology Studies at Leiden University in the Netherlands, is impressed with the new study’s sophisticated text-mining and language-processing techniques and methods. “I was pleasantly surprised to see that,” he says.

Brooks Hanson, executive vice president for science at the American Geophysical Union in Washington, DC, says the study uses good measures of quality. “If you’re going to do this at the scale that they’ve done it,” he says, “it seems a reasonable way to [ask]: Are many of the reviews that are being done helpful to the authors?”

Squazzoni acknowledges that in some areas of research, the developmental score rubric may not correspond well with the quality and thoroughness of reviews. For example, referees of theoretical papers are unlikely to mention data, and reviewers of papers in relatively new fields are less likely to mention past literature. “The problem with quality is the question of quality for what and for whom,” Squazzoni says. “Peer review involves at least three different actors: the authors, the reviewers, and the editors. If you zoom down at the microscopic scale of these people involved in the same process, there are very conflicting interests.”

For now, studies like this one must rely on the generosity of academic publishers in sharing their journals’ peer review data. In 2020 Squazzoni and colleagues called on journals, funders, and scholars to devise new ways of sharing peer review data to open up the reviewing process for scrutiny. But Squazzoni says little progress has been made to achieve that goal.