A published result marked “statistically significant” can look settled. When large teams repeated published experiments, many results came out weaker the second time, and many did not reach statistical significance. The same teams caution against reading that as proof that the original studies were wrong.
What a replication is
A replication repeats a published study with new participants, closely following the original method, and asks whether the original result appears again. One replication says something about one finding. The projects below repeated many findings in a coordinated way, so their results say something about how much weight a single published study can bear.
What three large projects found
The Open Science Collaboration completed 100 replications of psychology studies, by 270 contributing authors. Ninety-seven percent of the original studies had statistically significant results; 36% of the replications did. The authors write that if every original effect had been true and accurately estimated, they would have expected about 89 of the 97 replications of significant original results to succeed, and 35 did. Replication effects were, on average, half the size of the original effects.
Many Labs 2 ran preregistered replications of 28 classic and contemporary findings. Each was given to about half of 125 samples, which together included 15,305 participants from 36 countries and territories. Fifteen of the 28 replications (54%) found a statistically significant effect in the same direction as the original. Among the 26 findings measured with Cohen’s d, the median effect size was 0.60 in the original studies and 0.15 in the replications. Results varied more with the effect being studied than with the sample or setting, and whether the tasks were given in a lab or online accounted for very little of the variation.
Camerer and colleagues replicated 21 experimental social-science studies published in Nature and Science between 2010 and 2015, with samples on average about five times larger than the originals. Thirteen (62%) showed a significant effect in the same direction; by other measures of replication, the count ranged from 12 to 14. The replication effects were on average about half the size of the original effects. The authors also found that researchers’ beliefs about which results would replicate were strongly related to the outcomes, suggesting that the research community could predict which results would replicate.
The Open Science Collaboration project was supported by the Center for Open Science and the Laura and John Arnold Foundation; Many Labs 2 by the Center for Open Science and a grant through the Association for Psychological Science from the Laura and John Arnold Foundation.
Why a failed replication is not a verdict
The project authors are careful about what their numbers mean. The Open Science Collaboration warns that “[i]t is also too easy to conclude that a failure to replicate a result means that the original evidence was a false positive,” and that a successful replication does not by itself show that the theoretical understanding behind a finding is correct. Many Labs 2 notes that its 28 findings “were not a random sample of any definable population,” so its 50% rate under a strict criterion does not mean that half of all psychology findings would be reproduced, or fail to be reproduced, under similar circumstances, and that its failures to replicate “do not necessarily mean that the tested hypotheses are incorrect.”
The Open Science Collaboration puts the main point plainly: “a single study almost never provides definitive resolution for or against an effect and its explanation.”
What a p-value does not measure
All three projects counted a replication partly by whether it reached statistical significance, usually judged by a p-value. In 2016 the American Statistical Association published a statement setting out six principles. Among them: p-values “do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone”; a p-value, or statistical significance, “does not measure the size of an effect or the importance of a result”; and conclusions “should not be based only on whether a p-value passes a specific threshold.” The statement also says that proper inference “requires full reporting and transparency.”
Deciding the analysis in advance
Nosek and colleagues describe preregistration as defining “the research questions and analysis plan before observing the research outcomes.” It addresses the difference between testing a prediction made in advance and explaining a result after seeing the data. The authors write that “[m]istaking generation of postdictions with testing of predictions reduces the credibility of research findings.” All three replication projects above preregistered their replications.
INSTAR’s research interest
INSTAR’s interest is in the questions a careful reader can ask before relying on a single study. Has an independent team repeated it, and with how many participants? How large is the effect, and not only whether it was significant? Was the analysis planned before the data came in, and is the plan public? Do the authors report everything they measured?
These questions apply to any research that rests on experiments with people, including the topics on the psychology and cognitive sciences program pages. The brief on AI benchmark scores asks similar questions about a different kind of number, and the page on how this site checks a fact describes the checks applied to the sources cited here.