Vassar Scholars Examine Why Scientific Findings Don’t Always Hold Up
Are the results of experiments in the social and behavioral sciences published in scientific journals always reliable? A paper published in a recent edition of Nature magazine— conducted by hundreds of independent researchers, including Vassar Associate Professor of Cognitive Science Joshua de Leeuw and four of his students—suggests there is plenty of room for improvement in how findings about these experiments are reported.
The investigation, supported by the Systematizing Confidence in Open Research and Evidence (SCORE) program, showed that an expectation of getting the same results by re-running the same experiments cannot always be assumed. The SCORE researchers’ study on replicability—testing the same question using new data—found that the outcomes of the experiments were replicated only about half the time.
De Leeuw said that while this comprehensive study was groundbreaking, he was not particularly surprised by the results. “Science isn’t just facts and applying the scientific method to your experiments,” he said. “It is a human activity, and we humans are not perfect—and we inject our biases into our work.”
De Leeuw teaches his cognitive science students how to identify some of these biases. One of the most common ones, he said, is confirmation bias: “If someone tells you something you already believe to be true, you’re very likely to believe it, but if you are told something you don't agree with, you are more skeptical.
“I lecture about this in my Introduction to Cognitive Science class,” de Leeuw continued, “because I think it’s important. My research is all about building tools to make science more replicable; it’s kind of the science of doing science.”
One Vassar student who took part in the project in 2021/2023 academic year, Miles Bader ’24, said the sheer scale of the SCORE project lends credibility to the results. “With over 800 collaborators, this is a feat of coordination, delivering the sort of insights that wouldn't be available without that massive scale,” Bader said. “So, in many ways, it is humbling to be but one small contributor on a project of such massive proportions.”
Rachel Ostrowski ’24, who also took part in the research, called taking part in the project “the perfect introduction to academic research. The nature of the project—revisiting published research to attempt reproductions, replications, or generalizations—required us to think deeply about the process of doing research.”
The other two students who took part in the research were Tessa Charles ’22 and Meg Ritzau ’23.
Ostrowski said the spirit of the project never felt adversarial. “There seemed to be a collective understanding that everyone, including those whose work was being reviewed, stood to benefit from a more rigorous literature,” she said. “And that helped me understand why Josh has dedicated so much of his career to building these open-source experiment development tools: The best way to improve scientific practice, at scale, is to lower the barrier to doing it right.”