Physics reviewers are concise
Physicists can get wordy when describing their own work, but apparently they get straight to the point when providing feedback to others. According to a recent analysis of peer review reports by the firm Publons, scientists in the physical sciences generally write shorter reviews than their counterparts in biology, psychology, and the environmental sciences. Physicists’ propensity to keep it short applies to nearly every physics subfield and holds regardless of the prominence of the journal in which the research will appear. Although the company’s data set shouldn’t be considered a definitive representation of all academic reviews, it offers a rare glimpse at scientists’ reviewing habits.
Publons is a free-to-use website where researchers claim credit for conducting peer review on manuscripts. Users increase their “reviewer merit ” by recording information about their reviewing record and getting their reviews endorsed by peers. Peer review is usually a labor of love, and academics often have little incentive to do it apart from the fact that it enables them to keep abreast of new research in their area. Capturing reviewing history online helps academics get greater value out of their review work by enabling them to demonstrate to funders, during grant applications, the time they spend examining others’ research.
Since Publons launched in 2012, more than 300 000 researchers have signed up for the platform. The site earns revenue through partnerships with publishers and universities, and it has also received funding from angel investors in New Zealand. Last June it was acquired by the information firm Clarivate Analytics, which publishes journal impact factors .
Publons’s data set includes more than 300 000 reviews. The median review length for all fields is 292 words (see figure 1). Most of the physical sciences come in well below that number, including applied physical sciences/engineering, materials science, and general physics. Earth and space sciences are an exception, with a median review length of 494 words.
Figure 1. There’s a wide range of review lengths across the sciences, according to Publons reviewer-submitted data. The physical sciences tend to invite shorter reviews than other fields. Source: Publons
Digging deeper into the general physics category (see figure 2), reviews in nearly all subfields have a median length of less than 300 words. Andrew Preston, cofounder and managing director of Publons, suspects that is due to feedback that many authors receive prior to submission via arXiv. Nuclear physics is the outlier: The median of those reviews is 436 words, though nuclear is also the physics subcategory with the fewest reviews in the Publons database. “I can think of no conceivable reason why [nuclear physicists] would write longer reviews,” says Andrea Taroni, chief editor of Nature Physics in London.
Lester Ingber, principal investigator of the Extreme Science and Engineering Discovery Environment and Publons’s most highly rated nuclear physics reviewer by merit, says his latest review was around 400 words but adds that his reviews are usually around half that. “I don’t see the point of summarizing the paper, which I understand is what a lot of reviewers do as a way of demonstrating that they read and understood it,” he says.
Figure 2. Most physics-paper reviews are less than 300 words, according to Publons data. Nuclear physics is the notable exception, though it’s possible that a few long-winded reviewers skew the data. Source: Publons
AJ Mitchell, an experimental nuclear physicist at Australian National University in Canberra, says his reviews are typically between 500 and 800 words. Though he acknowledges that high-quality reports can be concise, he suspects that the small word counts “are potentially skewed by a large number of contributions from a small number of researchers.”
According to Publons data for all the sciences, high-impact journals tend to attract lengthier reviews than lower-impact publications (see figure 3). In a blog post summarizing the data analysis, the Publons team says this may be because reviewers hoping to publish in prominent journals in the future will likely turn in thorough reports to them.
Figure 3. Although the median review lengths for the most prominent physics journals exceed 350 words, there’s no obvious relationship between journal impact factor and review length in physics. There’s a clearer trend in science as a whole. Note that the x-axis shows ventiles and not actual journal impact factors; the range of impact factors in a given ventile for physics is different than that in the corresponding one for all the sciences. Source: Publons
Physics reviewers, however, buck that trend. Some higher-impact physics journals have relatively short reviews, on average, and some lower-impact journals have long reviews. “In my experience, physicists focus on the quality of manuscript first and foremost,” Taroni says.
Preston says he hopes Publons data will be used to build tools that improve the efficiency of peer review. He envisions, for example, tools that automatically alert journal editors about which referees are overloaded with review responsibilities. Last October, Publons launched the Review Distribution Index , a metric that quantifies reviewer workload.
Thumbnail image credit: the.Firebottle, CC BY-SA 2.0