How to Make Sense of Contradictory Science Papers – Issue 100: Outsiders
The science you can come across today can often appear to be full of contradictory claims. One study tells you red wine is good for your heart; another tells you it is not. Over the past year, COVID-19 research has offered conflicting reports about the overall effectiveness of wearing a mask. As scientists debate what policy best suits the current moment, they will be drawing on hundreds of studies; some that say masks are effective1 and some that say masks alone are not enough.2Naturally, given its outsized influence on society—especially during a pandemic—people tend to regard published science highly. This means that many of us expect scientists to be prudent in reporting their results. These ought to be true and justified by evidence, right? And surely, at a bare minimum, the researchers themselves ought to believe in what they are publishing, yes? Maybe not. The bar for publishing might, counter-intuitively, be lower than one might expect. “Scientific conclusions,” as we titled our recent paper,3 “need not be accurate, justified, or believed by their authors.”Why might it be worth worrying about how and when scientists decide to share their work? We’re not saying scientists generally lie about their published results (this has nothing to do with misconduct). Rather, we argue that scientific papers fulfill a useful social role by doing more than merely reporting on true discoveries. It’s enough for them to draw attention to an idea that is worth pursuing further—and an idea need not be true, well-justified given all our evidence, nor even believed by the scientist in order to pass that test. The peer-review process is, in fact, designed, not to detect fraud or data manipulation, but to select for what is noteworthy.4 What is considered unexpected and thought-provoking will not always track our all-things-considered judgments of what is true, but local community standards of best scientific practice having to do with how to go about data gathering and statistical testing.For scientists to collectively inquire effectively, they need to communicate interesting ideas to each other that are worth pursuing. Consider Avi Loeb, a theoretical astrophysicist who proposed the provocative hypothesis, not without some supporting data, that ‘Oumuamua wasn’t a comet but an alien light-sail. He, presumably, knew that more data would need to be gathered, and a more thorough study would need to be conducted, before the hypothesis could be justifiably believed.Nonetheless, perhaps it was appropriate for Loeb to publish his data and his hypothesis. He himself might even be agnostic toward the truth of the hypothesis. He likely knew that most of his colleagues would dispute his interpretation of the data, and with good cause. In spite of all this, it was still valuable for him to publicly communicate the possibility of a new hypothesis, because it can—and maybe actually did—spur more research into, and garner attention for, astronomy. Publishing those findings was not about communicating the truth but about saying that there is something exciting and interesting that requires further inquiry.Why might it be worth worrying about how and when scientists decide to share their work? The threat of misinformation spreading, you might say. But that isn’t all. As it stands, there is a mismatch between the rules scientists write by compared with those that laymen read by. And given that non-scientists are daily called on to make important decisions on the basis of scientific results, the potential for miscommunication makes possible momentous mistakes.For example, in the latter half of the 20th century, various linguistic theories were being proposed among academics about how children learn reading and writing. Public uptake of the whole language approach led to massive reforms in education curriculums in the United States that de-emphasized the role of phonics when teaching children how to read and instead emphasized the meaning of words in sentences. The linguistic theories that prompted these reforms were never empirically well-confirmed, and later research showed that teaching children systematic phonics is more effective.5 Education reformers treated on-going linguistic theorizing as if it were established fact.6You might see any one such case as being something between comic and tragic. But it is the collective impact of these errors that is of greater import. Each erodes our trust in science—but by just getting clearer on what scientific publication is generally for, we can stem some of that distrust.One study tells you red wine is good for your heart; another tells you it is not. The philosopher Karl Popper once said that science needs bold conjectures and attempted refutations. The idea now goes under the name of “falsificationism.” Being open to refutation is one of the most widely appreciated principles of science. But nowadays people tend to mostly emphasize the importance of refutation. We want to echo Popper and stress that bold conjectures are needed, too. And not just bold conjectures, but some means for scientists to advertise their best guesses. The publication system allows scientists to filter which conjectures are worth putting out there and taking seriously, without being so conservative as to prevent scientists from taking imaginative leaps.7In everyday life, we don’t often have need of this sort of conjectural laxity. If I ask you where you put the keys, I’m really not looking for your boldest theses on locksmithery. I really just want to know where the keys are. Such a humble purpose actually requires that, to be in good faith, my interlocutor meets a high conversational standard for assertion. Philosophers have explored what norms constitute such a standard.8These norms are the rules that assertions obey in well-behaved conversations. Some philosophers think you have to know that what you assert is true. Others think that, even if you don’t know it, you have to at least be justified in your claims. And others think that, even if you neither know nor have justification for your claims, you must at least be sincere and believe that you know it, or that it’s justified.Scientists need to communicate interesting ideas to each other that are worth pursuing. Fortunately, these norms of assertion don’t constrain science. We rightly tolerate the fact that it is, essentially, what you could politely call guesswork. Researchers must constantly be open to nature surprising them, and spread out over conceptual space, exploring whatever may be found. The trouble is that the general social esteem in which people hold science makes it natural for them to make an unhelpful assumption. That if scientific claims differ from the sort of claims each of us make every day, it is because the scientific ones have a better standing—better checked, have more evidence behind them, carry greater weight than our everyday assertions.But as we hope will be clear by now, this is far from the case. How disappointed will the consumer of science journalism thus feel if they pay attention to the modest track record of hot-off-the-press research!9 It would be no easy task to simply get scientists to work to a higher standard. Science actually requires a permissive attitude to published claims in order to do its job of discovering important truths.What is more, today, scientists often communicate results to each other outside of traditional publication venues. What should we make of the growing influence of “preprints” posted on online archives like medRxiv and bioRxiv? Science journalists are increasingly relying on papers in their reporting that aren’t peer-reviewed, changing the norms of communicating science. As a result, the general public is now engaging directly with COVID-19 research at an unprecedented rate.10 This also means scientists can learn about each other’s research faster and with more ease. They no longer need to go through a journal, and peer review, to communicate their novel, provocative ideas to each other.But this openness, of course, can also lead people to misunderstand what scientific papers are meant to communicate. These preprints are even further from being final, necessarily reliable discoveries. What we are witnessing is the process of science at play—the messy, inspired guesswork. We would do well to remember that. Loeb probably doesn’t seriously believe that alien tech careened through our solar system. But that provocation is nevertheless serious science.Haixin Dang is a postdoctoral research fellow at the University of Leeds. She currently works on an ERC funded project called Group Thinking, which explores collective attitudes toward the formulation of facts.Liam Kofi Bright is an assistant professor of philosophy at the London School of Economics. He works on the social epistemology of science, and generally would like to improve society somewhat.Footnotes1. Gandhi, M., Beyrer, C., & Goosby, E. Masks do more than protect others during COVID-19: Reducing the inoculum of SARS-CoV-2 to protect the wearer. Journal of General Internal Medicine 35, 3063-3066 (2020).2. Akhtar, J., et al. Can face masks offer protection from airborne sneeze and cough droplets in close-up, face-to-face human interactions?—A quantitative study. Physics of Fluids 32, 127112 (2020).3. Dang, H. & Bright, L.K. Scientific conclusions need not be accurate, justified, or believed by their authors. Synthese (2021).4. Nature editors. Can peer review police fraud? Nature Neuroscience 9, 149 (2006).5. National Reading Panel. Teaching children to read: An evidence-based assessment of the scientific research literature on reading. nih.gov (2000).6. Strauss, V. A case for why both sides in the ‘reading wars’ debate are wrong — and a proposed solution. The Washington Post (2019).7. We have our doubts about whether pre-publication peer-review is ultimately a good idea.8. Goldberg, S.C. Assertion: On the philosophical significance of assertoric speech. Oxford University Press, Oxford, U.K. (2015).9. Ioannidis, J.P.A. Why most published research findings are false. PLoS Medicine 2, e124 (2005).10. Fraser, N., et al. The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLoS Biology 19, e3000959 (2021).Lead image: Just Dance / ShutterstockRead More…
Read More
