The Test That Is 99% Accurate

The test is 99% accurate and it says positive. Is the patient 99% likely to be sick?

A 99% accurate test, a 1-in-10,000 disease, a positive result: about a 1% chance of disease. Count 10,000 people, and the positives come mostly from the healthy — about 100 false, 1 true. The trap: reading the test's accuracy as the answer, and forgetting how rare the disease was.

Every alert you will ever triage — fraud, an intrusion, a screening flag — is this test. When the thing you hunt is rare, even an excellent detector fires mostly on the innocent. Miss that, and a team learns the alarms mean nothing, then ignores the real one. See the base rate, and you know what a flag is worth before you act on it.

The real question: how likely is a positive to mean disease — not how likely a sick person is to test positive. That's the 99% on the box; flip the question, and the answer changes. Three terms: base rate, how common the condition is before any test (here, 1 in 10,000); false positive, a positive result on someone actually healthy; and the vertical bar, read "given" — P(A|B) is the chance of A, counting only cases where B happened.

Take 100 people who really are sick: 99 light up positive, one slips through — the test's hit rate on the sick, exactly as printed. Take 100 healthy people: 99% accuracy also clears 99 of them, but one lights up anyway — a false positive, one in a hundred. The obvious answer most people reach for: the test is 99% accurate, it said positive, so about 99% — most engineers guess above 90; in a classic 1978 study, most of sixty Harvard physicians and students guessed about 95% on a one-in-a-thousand version. But the 99% measured positive-given-sick. The question asked was sick-given-positive.

Picture all 10,000 people instead of one patient: on average, one is sick, the other 9,999 are healthy. Before anything runs — commit to a guess: of everyone whose test lights up, what share will actually be the sick one? Run it: the one sick square lights, and about 1% of the healthy — roughly 100 squares — light too. Pull every lit square into one pile: one sick, about a hundred healthy. The positive in hand is one of 101. About 1%.

That's Bayes' theorem, named: the chance of disease given a positive is the sick square's share of the whole pile — on top, how common the disease is times how often the sick test positive; underneath, that same piece plus the healthy share times the false-alarm rate. The base rate sits on top, with the same weight as the test itself. Plugging in the exact numbers gives about 0.0098 — the population count said 1 in 101 because it let the test catch the sick person for certain; both land at about 1%.

Before the test: 1 in 10,000. After a positive: about 1 in 100. The test did real work — almost a hundredfold — and the answer is still only about 1%. Both are true. The same shape shows up at hospital scale: one widely used sepsis model, checked externally, alerted on 18% of all patients and still missed two-thirds of actual sepsis cases.

Your Turn: Paste into Claude: "Take a test that catches 99% of sick people and clears 99% of healthy ones. Before computing anything, predict the base rate at which a positive result becomes a coin flip: a 50% chance of disease. Then compute the chance of disease given a positive at base rates of 1 in 100, 1 in 1,000, 1 in 10,000, and 1 in 100,000 — first by counting a population, then with Bayes' theorem. Finally, name one detector in my own field and the base rate it works against." What to look for: did the prediction come before the arithmetic, and do the counting and Bayes methods agree at every base rate?

Every number in this film traces to the chapter or is recomputed from its own inputs; constructed examples are labelled on screen. See the reel's FACTCHECK.md in the open course repository.

0:00 What this film is about 0:23 Why you should care 0:40 The question 0:52 Terms in this film 1:08 One positive 1:23 The sick 1:37 The healthy 1:53 The obvious answer 2:13 Ten thousand 2:28 Commit 2:40 The pile 2:53 Name it 3:13 Read the line 3:35 Back to the patient 4:00 Verdict 4:21 Your turn

Part of the Claude & Agentic AI series. Course: INFO 7375, Computational Skepticism for AI, Chapter 2, Northeastern University. @NikBearBrown

Bayes theorem explained, base rate fallacy, false positive rate explained, sensitivity specificity confusion, medical test accuracy misconception, base rate neglect, positive predictive value, INFO 7375, Northeastern computational skepticism, Bayesian reasoning example, conditional probability explained, sepsis alert false positive, fraud detection base rate, AI course lecture, probability of disease given positive test

What this video covers

• 0:00 — What this film is about 0:23 Why you should care 0:40 The question 0:52 Terms in this film 1:08 One positive 1:23 The sick 1:37 The healthy 1:53 The obvious answer 2:13 Ten thousand 2:28 Commit 2:40 The pile 2:53 Name it 3:13 Read the line 3:35 Back to the patient 4:00 Verdict 4:21 Your turn

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

What this film is about 0:23 Why you should care 0:40 The question 0:52 Terms in this film 1:08 One positive 1:23 The sick 1:37 The healthy 1:53 The obvious answer 2:13 Ten thousand 2:28 Commit 2:40 The pile 2:53 Name it 3:13 Read the line 3:35 Back to the patient 4:00 Verdict 4:21 Your turn

[0:00] Ahoy, this is Liam in for Bear with chapter 2 of info 7375 computational skepticism for AI at Northeastern. This film is about a test that is 99% accurate and a positive result that means about 1%. Count 10,000 people and the positives come mostly from the healthy, about 100 false, one true. The trap, reading the test accuracy as the answer and forgetting how rare the disease was. Why should you care? Every alert you will ever triage, fraud and intrusion, a screening flag is this test. When the thing you hunt is rare, even an excellent detector fires mostly on the innocent. Miss that and your team learns the alarms mean nothing then ignores the real one. See the base rate and you know what a flag is worth before you act on it. The real question, how likely is a positive to mean disease, not how likely a sick person is to test positive. That is the 99%

On the box. Flip the question and the answer changes. Three terms, base rate, how common the condition is before any test, here at one in 10,000. False positive, a positive result on someone who is actually healthy. And the vertical bar read given, P of a given B is the chance of a counting only the cases where B happened. One patient, a disease that about one person in 10,000 has. A test the company says is 99% accurate. The result comes back positive. So, how likely is it that this patient is sick? The rail at the bottom is where the answer goes. For now, it holds a question mark. First, what the 99% promises. Take 100 people who really are sick, one square each. Run the test, 99 light up positive. One slips through. That is the test's hit rate on the sick, 99 out of 100, exactly as printed. Now, 100 healthy people. 99% accurate also means

The test clears 99 of them. One lights up anyway, a false positive, one in 100, small. Remember that one. Real tests print two separate rates here. This example uses one number for both jobs. The obvious answer, the test is 99% accurate. It said positive, so about 99%. Most engineers say above 90. In a classic 1978 study, most of 60 Harvard physicians and students answered about 95 on a 1 in 1,000 version. But, look at what the 99 measured, positive given sick. We were asked the other way around, sick given positive. So, stop looking at one patient. Picture everyone who takes the test, 10,000 people, one square each. On average, one of them is sick. They're one square. The other 9,999 are healthy. Now, both facts we just watched can act on the whole crowd at once. Before anything lights up, commit. The test is about to run on all 10,000. The sick one will light. One

In 100 of the healthy will light, too. Of all the squares that light, what share will be the sick one? Pick a number. Lock it in. Run it. The sick square lights. Now, 1% of the healthy, about 100 squares scattered everywhere. Pull every lit square into the pen. One sick, about 100 healthy. The positive you are holding is one of 101, about 1%. What we just did deserves its name. The chance of disease given a positive is the sick square share of the pile. On top, how common the disease is times how often the sick test positive. Underneath, the whole pile, that same piece plus the healthy share times the false alarm rate. That is Bayes' theorem. And look where the base rate sits, on top with the same weight as the test. Read the line left, the chance of disease given a positive. On top, the base rate, 1 in 10,000 times the hit rate,

0.99. Underneath, that same piece plus the healthy share times the false alarm rate, 0.01. Every way to get a positive added up. Plug in the numbers, about 0.0098. The count said 1 in 101 because it let the test catch the sick person for certain. Both are about 1%. Back to our patient. Before the test, 1 in 10,000. After a positive, about 1 in 100. The test did real work, almost 100-fold, and the answer is still about 1%. Both are true. The same shape shows up at hospital scale. One widely used sepsis model, checked externally, alerted on 18% of all patients and still missed 2/3 of the sepsis. Not found here how common a disease must be before a positive is a coin flip. Let's recap with Claude, a 99% accurate test, a one in 10,000 disease, a positive result, about a 1% chance of disease. 99% is positive given sick. The question was sick given positive.

Count 10,000 people, one true positive, about 100 false. Bayes writes the count down and the base rate sits on top with the same weight as the test. Your turn, paste this into Claude. Take a test that catches 99% of sick people and clears 99% of healthy ones. Before computing anything, predict the base rate at which a positive result becomes a coin flip, a 50% chance of disease. Then compute the chance of disease given a positive at base rates of one in 100, one in 1000, one in 10000, and one in 100000 first by counting a population, then with Bayes' theorem. Finally, name one detector in my own field in the base rate it works against. Two things to look for. Did it lock a prediction before it computed? If it jumped to numbers, send it back. And did the counting and Bayes agree at every base rate? The test that is 99% accurate at Nick Berry Brown.