Unreliable Science

Scientists know that we should rarely draw conclusions about any scientific issue based on any single publication—the scientific process depends on replication and is iterative by design. To get to an “answer,” we have to look at the overall literature on a subject. Our understanding changes as knowledge accrues. We keep in mind that we do not know which findings will be replicated and which will, in the long run, prove to be cumulatively correct. 

How, therefore, do we distinguish, in the moment, the science we should believe? What evidence should we trust?

People trust different sources of evidence when establishing their beliefs. Some trust their feelings, others their observations, and some rely on the statements of respected authorities. But who is an authority? Because it is extremely difficult to judge the trustworthiness of most scientific claims, the average citizen can choose to dismiss announcements she prefers to forget or deny. And the average person can also be misled by “new information”; even “settled“ science can be shattered.

The association between childhood vaccines and autism, originally proposed in a single small study by a disgraced scientist, has been thoroughly debunked in more than twenty studies by different researchers in different countries. But the theory behind this spurious association has hung on. Beliefs, contrary to prevailing evidence, sometimes cannot be shaken. As a result, in this case, sentiment—the belief that our children have been harmed by science—influenced the selection of America’s new health secretary, who has now asked for the science to be reviewed yet again, and perhaps redone, looking for a different answer. 

The public’s judgment of science, in some sense, comes down to whom we trust.

The process of the health secretary’s selection revealed once again that the peer-reviewed scientific literature is a grab bag. Anyone can grab a paper from the international bag of published work and say, “Here’s my science about vaccines and autism. Prove me wrong.” Even though all published science is not created equal, there are more than three million “peer-reviewed” studies published annually, and the number is growing, straining the peer review process. New researchers, trained and willing and able to vet those articles, cannot keep up. Predatory, for-profit publishers then pump out research with insufficient scrutiny, leading to mass article retractions, while devaluing science and degrading our trust in scientists along the way.

So, the public’s judgment of science, in some sense, comes down to whom we trust. Since the science itself is too hard to understand without specialized knowledge, it is the messenger who generates trust in the science under discussion. Trust is a social judgment. We listen to scientists (or sometimes the representatives of science, such as political nominees) talk about science, and we judge their trustworthiness based on our perception of their expertise and their integrity. But similarly important are the messenger’s warmth, openness, and assertiveness, and our sense of their benevolence, whether their attitudes suggest self-interest or care for social benefit.

Reliable science that drives our health policy depends on reliable investigators, reliable peer review, and, more and more often, reliable messengers of science who do not misrepresent the data available. Scientists must not only do their science but also learn how to convey it in a way that is trusted by the public; falling short of doing so leaves open space for the specious inference that can be used to harm science and the public.

Previously in Observing Science: Null (And Void?)