What is the Base Rate Fallacy?

The base rate fallacy is judging what a test result means from how good the test is, while forgetting how rare the thing it looks for is. The base rate is that rarity: how many people have the disease before anyone is tested. Take a disease that 1 person in 1,000 has, and a test that catches every real case and wrongly flags 5% of healthy people. Line up 1,000 people. The one ill person is flagged. Of the 999 healthy people, 5% is about 50, and they are flagged too. So 51 people hold a positive result and one of them is ill: a positive is right about 2% of the time. The test is not bad. A 5% slip is small. It decides the answer only because there is one real case to find and 999 chances to slip, so the false alarms outnumber the one real case fifty to one. A 95%-accurate test for a 1-in-1,000 disease: a positive is right about 2% of the time. Count the crowd before you trust the signal.

The base rate fallacy is reading a result by the strength of the evidence alone and dropping the number that matters most: how common the thing was before the evidence arrived. That number is the base rate. In 2014, researchers at a Boston hospital put a question to 61 doctors and medical students. A disease affects 1 person in 1,000. A test for it catches every real case and has a false positive rate of 5%, meaning it flags 5% of people who do not have the disease. A person tests positive. What is the chance they have it? Count it in people. Of 1,000 tested, one has the disease and is flagged. The other 999 do not, and 5% of them, about 50, are flagged anyway. That is 51 positives, one of them real: about 2%. Fourteen of the 61 got it right, and 27 answered 95%. The same question in 1978 produced the same result: 11 right out of 60. What confuses people is that two percentages are both true. The test is wrong about only 5% of healthy people, which is modest. Yet 98% of its positives are wrong, because the healthy crowd is 999 times the size of the ill one, and a small slice of a huge crowd outnumbers the whole of a tiny one. The two scores of a test, sensitivity and specificity, cannot settle this on their own. They describe the test. What a result means for the person in front of you needs a third number, the base rate, and the figure that comes out of combining them is called the positive predictive value. A positive is not worthless. It moved this person from 1 in 1,000 to 1 in 51, twenty times the odds, which is why a positive screen is followed by a second, better test rather than a diagnosis. The same arithmetic runs through fraud alerts, security screening and any search for something rare: before trusting a signal, count the crowd it was pointed at.

The base rate fallacy is trusting a signal without asking how rare the thing it signals is, and the bank's fraud text shows the sum. Your card makes about 1,000 purchases a year, and suppose one of them is fraud. The bank's system catches that one. It also flags 2% of your honest purchases, about 20, usually a new shop or an odd hour. So 21 texts arrive asking "was this you?", and 20 of them are about your own shopping. The system is not broken. A 2% slip is small. The texts are mostly wrong only because there was one fraud to find and 999 honest purchases to slip on, so the slips outnumber the catch 20 to 1. The one fraud is the base rate, the 2% is the false positive rate, and the 21 texts are the positives. A text still matters: it moves the odds from 1 in 1,000 to 1 in 21, which is why the bank asks instead of blocking the card. Where the match ends: you can answer the bank in a second, while a patient with a positive result needs a second test.