You Tested Positive.
Should You Panic?
A test for a rare disease comes back positive. The test is 99% accurate. Most people assume that means a 99% chance of being sick. Almost nobody guesses right on the first try, including plenty of doctors.
Act one
Meet the population
A disease affects a small share of people. A test correctly catches 99% of true cases and correctly clears 95% of healthy people. Slide the disease's real rate and watch what a positive result actually means.
Out of 1,000 people, about 10 are sick and 10 test positive correctly. About 50 healthy people test positive anyway.
Act two
What if the test were better?
Keep the disease just as rare (1 in 100 people) and only change how good the test is at correctly clearing healthy people. Watch how much accuracy it takes before a positive result actually means something.
At 95% specificity and a 1% disease rate, a positive result is barely better than a coin flip.
The false alarms come from the crowd, not the disease
In a famous study, physicians at Harvard Medical School were given almost exactly this problem: a disease with a 1 in 1,000 prevalence, and a test with a 5% false positive rate. Most of them guessed the chance of actually having the disease after a positive result was around 95%. The correct answer was closer to 2%. Doctors, whose job is reading test results, got it wrong by a factor of nearly 50.
The reason has nothing to do with the test being bad. It's that healthy people vastly outnumber sick people when a disease is rare, so even a small false positive rate applied to a huge healthy population produces more false alarms than the tiny sick population produces true ones. The test's accuracy describes how it treats one person. The result you should believe depends on everyone else who could have triggered the same alarm.
Statisticians call this the base rate fallacy: ignoring how common something is in the first place, and judging a result only by the test's stated accuracy. The honest answer requires Bayes' theorem, and the number it produces is called the test's positive predictive value, the real-world reliability of a positive result once you account for the base rate.
- A spam filter that's 99% accurate still fills your inbox with false alarms if it scans a billion ordinary emails a day looking for a rare threat.
- Airport security screening for a rare event flags enormous numbers of innocent travelers for every real threat it catches, for exactly the same reason.