Reproduce or it didn’t happen: why replicable science is better science Premium

Reproducibility is the essence of scientific truth. Karl Popper wrote non-reproducible single occurrences are of no significance to science. To improve reproducibility, robust experimental design, better stats, data sharing, authenticated biomaterials, publishing negative data, better mentorship needed. We can't depend on one outcome/study. It's our collective responsibility to ensure reproducibility.

Since I was a little boy, like many Bengalis of my generation, I have been obsessed with Satyajit Ray’s tales about the mythical scientist Professor Shonku. Among his other magical inventions are “Miracurall,” a drug that cures all illnesses except the common cold; “Annihillin,” a pistol that can exterminate any living thing; “Shonkoplane,” a small hovercraft built on anti-gravity technology; and “Omniscope,” which combined the telescope, microscope, and X-ray-scope. Evidently, Prof. Shonku was a brilliant scientist and inventor.

Or was he?

The fact that none of Shonku’s powerful and useful inventions could be produced in a factory and that only he was capable of manufacturing them was a genuinely disheartening feature of his innovations. Later, after being exposed to the scientific community, I understood that Prof. Shonku couldn’t be considered a ‘scientist’ in the strictest sense of the word for this precise reason. The reproducibility of research is the essence of scientific truth and inventions.

In his 1934 book The Logic of Scientific Discovery, the Austrian-British philosopher Karl Popper wrote: “Non-reproducible single occurrences are of no significance to science.” This said, in some fields, especially observational sciences, where inferences are drawn from events and processes beyond the observer’s control, irreproducible one-time events can still be a significant source of scientific information, so reproducibility is not a critical requirement.

Consider the 1994 collision of Comet Shoemaker-Levy with Jupiter. It offered a wealth of knowledge on the dynamics of the Jovian atmosphere as well as preliminary proof of the danger posed by meteorite and comet impacts. One may recall the famous observation made by Stephen Jay Gould in his brilliant 1989 book Wonderful Life: The Burgess Shale and the Nature of History, that if one were to “rewind the tape of life,” the consequences would surely be different, with the likelihood that nothing resembling us would exist.”

However, scientists working in most disciplines do not have that kind of leverage, for sure. In fact, reproducibility – or the lack thereof – has become a very pressing issue in more recent years.

In a 2011 study, researchers evaluated 67 medical research projects and found that just 6% were fully repeatable whereas 65% showed inconsistencies when evaluated again. An article in Nature on October 12, 2023, reported that 246 researchers examined a common pool of ecological data but came to significantly different conclusions. The effort echoes a 2015 attempt to replicate 100 research findings in psychology, but managed to do so for less than half.