Skip to content
Mind Flaws

Judgment & Decision · Quick Test

Base Rate Neglect

A test came back positive: what is the chance the disease is real?

The experiment

Read the question and answer with the number you think is right.

A test detects a disease that affects 1 in every 1,000 people. The test has a false positive rate of 5%. A person took the test and the result was positive. Knowing nothing about their symptoms or signs, what is the chance that they actually have the disease?
%

What happened?

The most common answer is 95%, based on the idea that the test “is right 95% of the time”. But the disease is rare: out of every 1,000 people, only 1 has it. The other 999 are healthy, and 5% of them, about 50, get a positive result by mistake. Adding the sick person, who also tests positive, there are 51 positives, and only 1 of them is real: about 2%.

Why does your brain do this?

The question gives two numbers of different kinds. One is about the case, the test result; the other is about the population, how rare the disease is. The test result seems to say something about that person, while the share in the population seems like a distant fact. The mind leans on the first and sets the second aside, even though it weighs the most in the calculation.

When does this shortcut work well?

When what you are looking for is not rare, the accuracy of the test is close to the real chance. If half of the people tested had the disease, a positive would mean disease in about 95% of cases. Paying attention to new information also helps when the situation really has changed and the population average no longer describes that case.

Where does this show up in daily life?

In a routine screening for a rare disease, in a fraud detector that blocks legitimate purchases, or in a “suspicious behavior” alarm that goes off for almost everyone. Before taking an alert at face value, ask: out of every thousand cases, how many are real, and how many false alarms come along with them?

Deep Dive

Open Nerd Mode

The question in this experiment was asked in 1978 by Ward Casscells, Arno Schoenberger and Thomas Graboys to 60 people at Harvard Medical School: 20 students, 20 residents and 20 attending physicians. Only 11 (18%) got it right, and 27 (45%) answered 95%.

In 2013, Arjun Manrai and colleagues asked the same question to 61 physicians and medical students at a hospital in the Boston area. 14 (23%) got it right, a difference that was not statistically significant compared with 1978. The most common answer was still 95%, given by 27 people (44%), and the median answer was 66%, 33 times the correct value. The authors assumed a test that finds every sick person and arrived at 1.96%. They call for more training in statistics throughout medical education.

The phenomenon was described in the 1970s by Daniel Kahneman and Amos Tversky, who studied how people make predictions, and became known as the base-rate fallacy through the work of Maya Bar-Hillel (1980). The “base rate” is the share of cases in the population, the probability before any test. The correct calculation uses Bayes’ theorem, which combines that share with the quality of the test.

The evidence badge is “mixed” because the basis here is two small samples of doctors in the Boston area, with a single question. The result held up 35 years apart, but it does not show how much the error changes with other questions and other groups of people.

Technical terms

  • Probability
  • Conditional probability
  • Base rates

Discovery

Described by Daniel Kahneman e Amos Tversky in 1973.

Level of evidence

Mixed evidence

The effect exists, but its size, conditions or interpretation vary across studies.

Basis: Manrai, A. K.; Bhatia, G.; Strymish, J.; Kohane, I. S.; Jain, S. H. (2014). Medicine’s uncomfortable relationship with math.

References

  1. Original studyCasscells, W.; Schoenberger, A.; Graboys, T. B. (1978). Interpretation by physicians of clinical laboratory results. New England Journal of Medicine, 299(18), 999–1001.
  2. ReplicationManrai, A. K.; Bhatia, G.; Strymish, J.; Kohane, I. S.; Jain, S. H. (2014). Medicine’s uncomfortable relationship with math. JAMA Internal Medicine, 174(6), 991–993.
  3. Original studyKahneman, D.; Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4), 237–251.
  4. Original studyBar-Hillel, M. (1980). The base-rate fallacy in probability judgments. Acta Psychologica, 44(3), 211–233.

Review

This text has not been reviewed by a volunteer yet. Want to help review it?

Found a mistake? Write to [email protected]