What is Statistical Power?

Statistical power is the chance that a study will detect an effect that really exists, such as a drug lowering blood pressure by a few points. Four things set that chance before the study starts: how many people are measured (the sample size), how big the effect is, and how much the measurements vary from person to person. The fourth is how strict the cutoff is for declaring a result real. More people, a bigger effect, less variation and a looser cutoff all push the chance up. The rulebook for drug trials, the ICH E9 guideline, tells researchers to accept a 10% to 20% chance of missing a real effect. A trial that accepts 20% has 80% power: of five real effects, it finds four and misses one, by design. So a low-powered study that finds no effect has not shown there is none. The question to ask of any "no effect found" is how good its chances were.

Statistical power is the probability that a study will find an effect that is really there. The effect is the difference being hunted: a drug lowering blood pressure by 5 points, say, against a dummy pill. Finding it means the result clears the cutoff for being called statistically significant, usually that chance alone would produce a difference this large less than 5% of the time. Missing a real effect is a false negative, and power is the chance of avoiding that miss. Four inputs set it. First, sample size: 400 patients give a better chance than 40, because the average of many is steadier. Second, effect size: a 5-point drop is easier to see than a 1-point drop. Third, variability: if blood pressure bounces around from patient to patient, a real 5 points hides in the scatter. Fourth, the cutoff: a stricter pass mark means more real effects fail it. The ICH E9 guideline for drug trials tells researchers to accept a 10% to 20% chance of a miss, so power of 80% to 90% (100 minus 20, 100 minus 10), and to recruit enough people to reach it. Why not 99%? Because finding an effect half the size takes roughly four times as many participants. A 2013 review estimated power across neuroscience studies at between about 8% and 31%, so most would miss a real effect more often than they found it. It also found a hidden cost: the effects such studies do find come out exaggerated, because only the readings that overshot by chance cleared the cutoff. So a null result needs a question put to it: what size of effect could this study have found? A found effect needs a different one: how big is it, which is effect size, not power. Power is whether a study can see the difference, not how much it matters.

Statistical power is the chance a study finds an effect that is really there, and a search party for a lost dog works the same way. Three neighbours walk the woods for an hour and come back with nothing. Thirty people search until dark and come back with nothing. Only the second nothing tells you much about whether the dog is in those woods. Size matters too: three people would spot a Great Dane, while thirty might walk past a dachshund lying still in tall bracken. The searchers and their hours are the sample size, the dog's size is the effect size, the bracken and the failing light are the variability, and the rule for calling it found is the cutoff. A thin search that finds nothing tells you about the search, not the dog. The match ends here: a dog can wander off and an effect cannot, and the searchers eventually learn whether the dog was there, while a study never does.