Science

tags
Empiricism

ERROR: “a bug bounty program for science”

Replication Crisis

There is some argument as to whether there is a “replication crisis” happening, particularly in biology and psychology (and economics, if you call that a science). Many attempts to replicate fairly widely trusted experimental results have resulted in failure. A landmark paper in this regard is Ioannidis's Why Most Published Research Findings Are False.1 There has been some pushback against Ioannidis's exact calculations, but his proposed reforms are pretty widely believed to be good ideas.

It isn't necessarily surprising that replication is uncommon. Bird argues that because a hypothesis is a priori far more likely to be false than true, we should expect significant results not to replicate unless they have very high statistical power, and to expect otherwise is to commit the base rate fallacy.2

Regardless, there are some explanations offered for the crisis that probably do contribute and probably should be fixed: secret data, data dredging, low statistical power, publish or perish, publication bias, etc. Some solutions are study preregistration, result-blind review, and higher p-value standards.

Footnotes:

1

John P. A. Ioannidis, “Why Most Published Research Findings Are False,” Plos Medicine 2, no. 8 (August 2005): e124, https://doi.org/10.1371/journal.pmed.0020124.

2

Alexander Bird, “Understanding the Replication Crisis as a Base Rate Fallacy,” The British Journal for the Philosophy of Science 72, no. 4 (December 2021): 965–93, https://doi.org/10.1093/bjps/axy051.