Missing Is Not One Thing

Two columns are each 18% blank. Why is one safe to fill in and the other a trap?

The reason a value is missing decides whether filling it in is safe. Sort the rows by what you can see: some blanks line up. Some never will. The trap: the worst kind of missing looks exactly like the harmless kind.

Every dataset you inherit has blanks, and the first fix anyone reaches for is the average. When the reason is harmless, the average lands on the truth. When the people who left it blank are different, it drifts in a direction you can't measure. Knowing the three kinds tells you which question no plot can ever answer for you.

Three terms: a missingness matrix — every row a thin line, every column a stripe, dark where a value is absent. Impute — fill a blank with an estimate, like the average of what's visible. A subgroup — rows sharing a trait, like everyone under 25.

Two tables of the same fifty people, each with an income column 18% blank — nine rows out of fifty, scattered in both, identical count and look. Fill both with the visible average. One fill lands on the truth. The other lands somewhere else. A third, easier table shows the baseline: nine income cells wiped completely at random, filled with the visible average — the fill lands right on the true average, because random loss shrinks a sample without tilting it.

Table A behaves differently: the same fill lands too high. Sorting by age reveals why — every blank row belongs to someone under 25, and in this table the young earn less. Filling each blank with its own age group's average slides the estimate right back onto the truth. Doing exactly the same thing to Table B, whose blanks look just as scattered, barely moves the estimate at all — it stays low, and nothing in the table explains why.

The move: stop reading the matrix in file order. Sort by a column you can see and watch where the blanks go. Table A's blanks slide together under age. Table B's stay scattered under every visible column — age, city, anything recorded. The one column Table B doesn't have is the true, unwritten income value itself. Predicting before sorting: Table B's nine blanks turn out to cluster at the very top — the nine highest earners — while a genuinely random table's blanks stay scattered. From inside the visible data, those two cases looked identical.

These three patterns have names, from Donald Rubin's work on missing data: Missing Completely at Random, where the chance of a blank depends on nothing; Missing at Random, where it depends on something visible, like age; and Missing Not at Random, where it depends on the missing value itself. A sort can reveal the first two. The third, no sort ever will — the classic example is a study's final measurement, blank for patients who left early, where the sort that would line things up is exactly who got sicker, the very thing that's missing. You don't find this kind with a plot; you find it by asking who would not report this, and why.

Your Turn: Paste into Claude: "A clinic dataset has a column pain_score that is 18% blank, and the missingness matrix shows the blanks cluster in evening appointments. Before writing any code, predict: MCAR, MAR, or MNAR? Then give me one story under which it really is MAR, and one under which it is actually MNAR. For each story, say which way an average fill would move the mean pain score, and name one question I should ask the clinic that no plot of this data could answer." What to look for: did it predict before reaching for code, and does its MNAR story name who would not report, and why, rather than just recommending a fancier fill.

Every number in this film traces to the chapter or is recomputed from its own inputs; constructed examples are labelled on screen. See the reel's FACTCHECK.md in the open course repository.

0:00 What this film is about 0:20 Why you should care 0:37 The question 0:48 Terms in this film 1:03 Two tables 1:19 Random wipe 1:39 Table A 1:56 Age group fill 2:12 Sort the rows 2:29 Predict 2:43 The hidden sort 2:53 Three names 3:11 Who would not report 3:27 Payoff 3:46 Verdict 4:04 Your turn

Part of the Claude & Agentic AI series. Course: INFO 7375, Computational Skepticism for AI, Chapter 3, Northeastern University. @NikBearBrown

missing data types, MCAR MAR MNAR explained, missing completely at random, missing at random, missing not at random, Rubin missing data classification, missingness matrix, data imputation bias, INFO 7375, Northeastern computational skepticism, mean imputation bias, subgroup imputation, data science missing values, statistical bias missing data, AI course lecture

What this video covers

• 0:00 — What this film is about 0:20 Why you should care 0:37 The question 0:48 Terms in this film 1:03 Two tables 1:19 Random wipe 1:39 Table A 1:56 Age group fill 2:12 Sort the rows 2:29 Predict 2:43 The hidden sort 2:53 Three names 3:11 Who would not report 3:27 Payoff 3:46 Verdict 4:04 Your turn

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

What this film is about 0:20 Why you should care 0:37 The question 0:48 Terms in this film 1:03 Two tables 1:19 Random wipe 1:39 Table A 1:56 Age group fill 2:12 Sort the rows 2:29 Predict 2:43 The hidden sort 2:53 Three names 3:11 Who would not report 3:27 Payoff 3:46 Verdict 4:04 Your turn

[0:00] Vanacam, this is Liam in for Bear with chapter three of Info 7375 computational skepticism for AI at Northeastern. This film is about blank cells. The reason a value is missing decides whether filling it in is safe. Sort the rows by what you can see and some blanks line up, some never will. The trap, the worst kind of missing looks exactly like the harmless kind. Why should you care? Every data set you inherit has blanks and the first fix anyone reaches for is the average. When the reason is harmless, the average lands on the truth. When the people who left it blank are different, it drifts in a direction you cannot measure. Knowing the three kinds tells you which question no plot can answer for you. The real question before I fill them in, should I ask why it is missing? How much is missing is identical here, 18% both times. The count cannot tell them apart, the reason can. Three words you'll

Hear, a missingness matrix, every row a thin line, every column a stripe, dark where a value is absent. Impute, fill a blank with an estimate such as the average of the values you can see and a subgroup, rows that share a trait such as everyone under 25. Two tables of the same 50 people. In each, the income column is 18% blank, nine rows out of 50. Look at the dark cells, scattered in both, same count, same look. Fill both with the average income you can see. One fill lands on the truth, the other lands somewhere else. Which is which? Start with the easy case, a third table where nine income cells were wiped at random, a coin flip, nothing to do with the person. Fill the nine with the average of the 41 you can see. We built this table so we can reveal the true average. The fill lands right on it. Wipe a different nine at random, it lands on

It again. Random loss shrinks your sample, it does not tilt it. Now table A, same fill, same average of the visible rows. This time the dot lands to the right of the truth, too high. Something about who is blank pulled it. Look along the age stripe beside each dark cell. Every blank row belongs to someone under 25. In this table, the young earn less and they are the ones who skipped the question. The natural fix, fill each blank with the average of its own age group. Do it for table A and the dot slides back onto the truth. Now do exactly the same thing for table B whose blanks look just as scattered. The dot barely moves. It stays low and nothing in the table tells you why. Here's the move, stop reading the matrix in file order. Sort the rows by a column you can see and watch where the blanks go. Sort table A by age, the nine dark cells

Slide together all inside the under 25s. Sort table B by age, still scattered. By city, scattered. A block means the blanks depend on something you can see. Table B stays scattered under every column it has. So here is the one column it doesn't have, each person's true income, the value that was never written down. Before I sort by it, commit. Where will table B's nine blanks go in the random tables? Sort table B's blanks slide into one block at the very top, the nine highest earners. The random tables stay scattered. From inside the data, those two looked identical. These deserve names and they come from Donald Rubin's work on missing data. Missing completely at random, the chance of a blank depends on nothing. Missing at random, it depends on something you can see like age. Missing not at random, it depends on the missing value itself. The first two, a sort can show you. The third, no sort you can run ever

Will. Same shape, different table. A study's final measurement blank for the patients who left before the end. Sort by anything recorded, scattered. The sort that would line them up is who got sicker, which is the very thing that's missing. So you don't find this kind with a plot. You find it by asking who would not report this and why. Back to the two tables, both 18% blank, both scattered in file order. Sort by age, table A lines up so fill within the age group and you land on the truth. Table B never lines up and its average sits low, biased in exactly the direction you cannot measure. Not taught here, how to choose an imputation method, or rows are joined, dropped before any matrix could see them. Let's recap with Claude. Missing is not one thing. If the blanks depend on nothing, an average fill lands on the truth. If they depend on something you can see like age, sort by it,

Fill within the group, and you recover it. If they depend on the missing value itself, every fill drifts and no plot can show you. For that kind, ask who would not report and why. Your turn. Paste this into Claude. A clinic data set has a column pain score that is 18% blank, and the missingness matrix shows the blanks cluster in evening appointments. Before writing any code, predict MCAR, MAR, or MNAR. Then give me one story under which it really is MAR, and one under which it is actually MNAR. For each story, say which way an average fill would move the mean pain score. And name one question I should ask the clinic that no plot of this data could answer. Two things to look for. Did it commit to a prediction before it reached for code? And does its MNAR story name who would not report and why, instead of recommending a fancier fill? Missing is not one thing at Nick Bear Brown.