What is Publication Bias?

Publication bias is the tendency for studies that find an effect to get published while studies that find nothing stay unpublished, so the published record makes effects look more real and larger than they are. Nobody has to cheat. Journals prefer results, researchers know it, and null findings never get written up. Anyone reading only what is published sees the wins and few of the losses. The older name for it is the file drawer problem.

Publication bias is a distortion in the published research record that arises because studies with positive results are more likely to be published than studies that find no effect. The mechanism runs on ordinary incentives rather than fraud. A trial showing a drug works gives readers something to use and cite, so a journal accepts it. A trial showing the drug does nothing reads as no story, so it is rejected or never submitted. Researchers learn this and stop writing up null results, and those studies go in a drawer: the older name is the file drawer problem. The reader of the literature then sees a biased sample. If ten trials of a treatment were run and the two positive ones were published, the treatment looks proven when eight out of ten found nothing, and no single paper had to be wrong. The bias is hard to see because the missing studies are missing: nobody can count what was never published. The main defence is to make them countable. Registering a trial before it starts leaves a record of every study attempted, so anyone reviewing the evidence later can see how many trials were run and ask where the rest went.

A friend who plays poker every weekend tells you about the nights he wins. Big pot, clever bluff, walks home with the table's money. He never mentions the other nights, and not because he is lying: nobody tells a story about losing forty dollars quietly and going to bed. Listen to him for a year and you would back him against anyone. His actual record is close to break-even. The winning nights are real. It is the missing nights that fool you, and you cannot see them because they were never told. Published research works the same way, and the missing nights are the studies that found nothing.