Evaluating Scientists

The work of science is generally carried out in institutions that create the conditions for research and scholarship to flourish. Those institutions, principally universities—but also including research institutes, private corporations, and some other entities—have created structures and mechanisms to evaluate the work of scientists. At universities—where in the U.S. most of the work of science is done—scientists are typically professors who advance through the ranks as assistant, associate, and full professors. Institutional review committees evaluate the contribution of the scientist to move them along the path to promotion.

How does one best evaluate the contribution and work of scientists? At some level, the answer is easy and obvious: we consider the production of the scientist, typically measured in peer-reviewed papers (and to some extent, for many fields in the life sciences, grant funding). We can count how many papers a scientist has published and, adding nuance, papers published in more prestigious journals. But what good do papers do if they are not read? Is a scientist who publishes 1,000 scarcely read papers more meritorious of advance than a scientist who has published 10 widely read papers?

Typically, this question is overcome in the field by counting the number of citations, that is, the number of times that a particular paper has been cited by other scientists. There are variants of this approach, including h-index and i-10 index. These all are fundamentally a way of quantifying the penetrance of a peer-reviewed paper in the scientific literature, and therefore of evaluating the contribution a scientist has made. Such systems have limitations of course: how to consider the role of an individual in a multi-author publication, or the inclusion in such scorings of review articles, which are not original research. So, to augment the evaluation, institutions get letters of input on a scientist’s work when they are up for promotion, asking other scientists about their impression of the quality that work and the importance to the field of study, a form of assessment that is added to the quantitative indices.

There is clearly no right answer to how we measure the impact of science, or the work of a particular scientist.

This is all to the good. But, as the world evolves, this is also where matters get tricky. Because these metrics all refer to the weight given to the work of science within science. But is that the only metric that we should use to evaluate science? What about measures that can point to impact in the world? What about uptake of ideas by policymakers and communities? 

There are new measures that have been proposed, to be sure, including, for example, Altmetric scores, which aim to quantify the attention papers have received by non-academic venues, such as social media platforms. But does it matter if a particular peer reviewed paper’s content is picked up by The New York Times? Does it matter if it is featured on a podcast? Often, scientific articles with shock value that speak to a fad of the moment receive media attention although their findings fade quickly, and how should we evaluate those? After all, nothing gets media attention like research about pets. Does a piece of research matter more if it becomes the foundation for legislative effort? Newer databases like Overton offer an opportunity to search for mention of academic papers in policy documents. Is this a better measure of the impact of science?

There is clearly no right answer to how we measure the impact of science, or the work of a particular scientist. And we are well aware that the impact of a scientist may not be recognized in their time. One needs to think only of Ignaz Semmelweis, whose contributions to the recognition of the importance of hand-washing were seminal—and who was roundly castigated and ignored in his own time—to recognize that important work may take decades to emerge as such. That, of course, does not change the import of evaluating scientists during their careers. 

As the world of science moves beyond the comfortable insularity of a hermetic science-only-for-scientists focus, it is becoming ever more important to think of ways in which we can evaluate the full contribution of science, to the field, and to the world that supports the field. In the short term, an awareness of the ways in which a fuller picture of the work of an individual scientist may matter seems central to ensuring that we recognize and reward work that will matter in the decades to come.

Previously in Observing Science: Trust and Verify