It's Time To Talk About Peer Review

Fifty, or even twenty-five years ago, procuring, analyzing, and presenting data in a way that illuminated larger truths was the exclusive province of scientists who did their work in institutions and then published their work in academic journals. That began changing with the democratization of data, more ready access to electronic means of disseminating writing, and with the rise of fields like data journalism, which started doing much of what scientists did—evaluated data and presented it in a way that produced meaning. These evolutions are, in the main, positive. They allow greater access to data for many well outside the confines of mainstream science, encouraging transparency. They allow for the possibility of insights that might otherwise have been missed by scientists and create ample space for publishing these same insights. In addition, these shifts have allowed for the more rapid analysis of data to address pressing contemporary issues, unbound by the processes of science.

What then continues to set science apart? While one may be tempted to lean on rigor, that is not necessarily an argument that those in science would want to get into, pitting their rigor against, say, those of data journalists working in organizations that themselves take pride in their thoroughness and objectivity. We suggest that the key distinguishing feature of science, rather, is peer review. While anyone can analyze data, make it available, even publish it in a leading magazine, only science has a rigorous system that regulates what gets published in its mainstream journals through a filter of peers who know the state-of-the-field and methods being used, and in written reviews offer comments and request improvements, and therefore serve as arbiters of quality external to the scientists.

It is the process of peer review that distinguishes science from any other form of representation of analysis. The process is, of course, imperfect. Weak and problematic papers certainly get published. Expert reviewers have their own biases. But peer review has given a legitimacy to science and as such the process is worth contemplating and strengthening. At this moment, the state of peer review is not strong. Peer review has always depended largely on the good will of scientists who volunteer their time to examine and appraise the work of their peers, seeing this as part of the business of the field, a professional duty.  Leaving aside the oddity of volunteer labor by scientists for products published by for-profit journals, this approach has always depended on scientists wanting to peer review. And that largely held, until COVID-19.

During COVID-19, a combination of general retrenchment of work habits across sectors, including science, a growth in papers written and needing peer review, and general uncertainty about the state of the world resulted in enormous peer-review backlogs. In part in response to this, systems proliferated that made it easier for editors to identify potential reviewers, increasing the volume of asks, even as the “yield” on those asks (i.e., scientists who agree to perform the service of peer review) decreased.

The “experts” who end up reviewing may not be experts, and may be merely those available to do the job. This has created a breakdown in the peer-review system like none in lived experience. And that breakdown has significant implications given the centrality of peer review to the entire enterprise. Would science be the same without the central role played by peer review? Is it time, we wonder, for a wholesale re-imagination of what peer review might look like? Of the rewards associated with it? Of the role of journals, and how they can be re-configured? We address some of these ideas in subsequent essays.

Previous Issue: Funding and the Questions We Ask