What is the Replication Crisis?

The replication crisis is the discovery that a large share of published studies do not give the same result when other researchers run them again. That second run is a replication: the same method, new participants, another team. Any single study can land on a fluke, so one run is never proof. How many flukes reach print depends on how a field works. Small samples, analysis chosen after seeing the data, and journals that publish the surprising positives and not the dull nulls all let more through than chance alone would. In 2015 a team of psychologists reran 100 published studies from their own field. The originals had been statistically significant 97 times out of 100. The reruns were significant 36 times, and the effects they found were about half the size. A failed rerun does not show the first result was false. It shows one run was weak evidence. So the question for any "a study finds" headline is whether anyone has found it twice.

The replication crisis is the finding, made when researchers began redoing published studies in bulk, that many do not hold up. A replication is a second, independent run: another team follows the original method with new participants and sees whether the result returns. For decades this was rare. Journals wanted new findings rather than repeats, and careers were built on discoveries, so the check that research relies on was seldom carried out. Any single study can hit a fluke, so one run is never proof. How many flukes reach print depends on three habits. A small sample swings by chance, so a small study is wrong more often. Choosing how to analyse the data after seeing it lets luck pick the method, which is p-hacking. And journals print the surprising positives while studies that found nothing stay unpublished, which is publication bias. In 2011 a group of 270 researchers began the largest such check. They reran 100 studies published in three psychology journals in 2008, finished in 2014 and reported in 2015. Of the originals, 97 had been statistically significant. Of the reruns, 36 were, and the effects measured were about half the size of the originals. Which results held up was predicted by the strength of the first result, not by who ran the rerun, and the most surprising findings held up least. The authors added a caution: a failed rerun does not mean the original was a false positive, because a second run can fail by chance or because a detail of the method mattered. It means one study is weak evidence. The fixes since: write the analysis down before collecting data (preregistration), recruit more participants, let journals accept a study on its design before any results are in, and share the data. For a reader, the test of any finding is whether someone else has got it again.

The replication crisis is what a field finds when it reruns its published studies and many do not come out the same, and a first night at bowling shows why one result is not a level. Someone bowls for the first time and scores 180, and the group talks about nothing else. Over the next ten weeks they average 110. The 180 was real: the pins fell. It was also one game, and one game mixes skill with luck, so it was a poor guide to how good the bowler is. Only the repeats showed that. A league that records first games and never the repeats would have a book full of 180s, and its members would look far better than they are. A field that publishes first results and rarely reruns them has that book. The low repeats do not make the 180 a lie, and a failed replication does not make the first study false. The match ends there: a bowler can rerun for the price of a lane, and an experiment can cost years, which is why so few were rerun.