Five Things the Model Can't Do for Itself
You might ask what kind of person should supervise an AI. The chapter says that is the wrong question: not personality traits, not vibes. The question is what a supervisor can do that the system cannot do for itself. A short course film for INFO 7375 Computational Skepticism for AI (Northeastern University, Nik Bear…
Five Things the Model Can't Do for Itself
You might ask what kind of person should supervise an AI. The chapter says that is the wrong question: not personality traits, not vibes. The question is what a supervisor can do that the system cannot do for itself. A short course film for INFO 7375 Computational Skepticism for AI (Northeastern University, Nik Bear Brown) on the Five Supervisory Capacities from Chapter 1. Verification is the problem; the supervisor stands between the model and the world; and the chapter counts five things the supervisor does there, each with its own quiet failure. Plausibility auditing: the clinician's twitch at a low triage score, a check against a prior the model does not have. Problem formulation: the triage system answered "what statistical category is this?" when the clinical question was "how likely is a life-threatening condition?" Tool orchestration: picking the AI, human judgment, a different model, or no tool yet, and the mandate to say "not here." Interpretive judgment: the chapter's example of a 5% false-positive rate, fine for a spam filter and catastrophic for a cancer screen. Executive integration: many outputs made into one decision that someone stands behind, the rarest capacity and the one most often missing. Where you cannot name who exercises a capacity, you have found a gap, and in the chapter's words, gaps are where the patients die.
Liam, in for Bear.
Course: INFO 7375 Computational Skepticism for AI · Northeastern University · Nik Bear Brown
Chapters
0:00 The question: what can a supervisor do that the AI can't? 0:18 Four terms: capacity, prior, formulation, integration 0:36 Verification is the problem; the supervisor between model and world 0:50 One. Plausibility auditing: the clinician's twitch 1:06 Training data vs the world; the score that sails through 1:20 Two. Problem formulation: the question never asked 1:37 Three. Tool orchestration: AI, human, another model, or no tool yet 1:50 "Not here" is supervisory work; used because it is available 2:05 Four. Interpretive judgment: 5% at a spam filter vs a cancer screen 2:22 Five. Executive integration: where the buck stops 2:37 The rarest capacity; decisions nobody owns 2:54 Name the capacity at every step; the gap 3:08 Your Turn
YOUR TURN
Pick one AI deployment you know, or one step of one, and paste this into Claude:
Here is one AI deployment I know, or one step of it: describe it. For each of the five supervisory capacities (plausibility auditing, problem formulation, tool orchestration, interpretive judgment and executive integration), write who exercises it, in one line each, or write "nobody." Then tell me which line you are least sure of, and why.
Then check two things yourself:
- Is every "who" a named role, not "the team"?
- Did you write "nobody" at least once, honestly?
Sources
- Nik Bear Brown, INFO 7375 Computational Skepticism for AI, chapter 1, The Skeptic's Toolkit, section "The Five Supervisory Capacities": the frame (capacities, not traits), the five capacities with their examples (the 49-year-old with the swollen leg, the two triage questions, "no tool yet", the 5% false-positive rate, the buck stops), the table's failure column, and the closing paragraph ("gaps are where the patients die").
The film states nothing beyond the chapter's section; the one number shown, 5%, is the chapter's illustrative example, not a measurement.
@NikBearBrown
Part of the Claude & Agentic AI series. Made with the brutalist.art toolkit: Kokoro for the voice, Manim for the drawings.
▶ The full series playlist — Brutalist https://www.youtube.com/playlist?list=PLG9H-C6rp5RU
What this video covers
- 0:00 — The question: what can a supervisor do that the AI can't?
- 0:18 — Four terms: capacity, prior, formulation, integration
- 0:36 — Verification is the problem; the supervisor between model and world
- 0:50 — One. Plausibility auditing: the clinician's twitch
- 1:06 — Training data vs the world; the score that sails through
- 1:20 — Two. Problem formulation: the question never asked
- 1:37 — Three. Tool orchestration: AI, human, another model, or no tool yet
- 1:50 — "Not here" is supervisory work; used because it is available
- 2:05 — Four. Interpretive judgment: 5% at a spam filter vs a cancer screen
- 2:22 — Five. Executive integration: where the buck stops
- 2:37 — The rarest capacity; decisions nobody owns
- 2:54 — Name the capacity at every step; the gap
- 3:08 — Your Turn
Transcript
A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.
The question: what can a supervisor do that the AI can't?
[0:00] Ciao, this is Liam in for Bear with a short film for info 7375 computational skepticism for AI. You might ask what kind of person should supervise an AI. The chapter says that is the wrong question. Not personality traits, not vibes. The question is what a supervisor can do that the system cannot do for itself.
Four terms: capacity, prior, formulation, integration
[0:18] Four terms: capacity, something the supervisor can do that the system cannot do for itself. Prior, the sense built over years of what kinds of cases tend to be what kinds of cases. Formulation, the exact question the system is being asked to answer. Integration, many outputs made into one decision that someone stands behind. Verification is
Verification is the problem; the supervisor between model and world
[0:37] The problem. A model makes an output and someone has to check it against the world. That someone is the supervisor standing between the model and the world. The chapter counts five things the supervisor does there that the model cannot do for itself.
One. Plausibility auditing: the clinician's twitch
[0:50] One, plausibility auditing. The chapter's case, a low triage score for a 49-year-old with a swollen leg. An experienced clinician looks at it and something twitches. The twitch is the audit, a check against a prior, the sense built over years of what kinds of cases tend to be what. The model cannot
Training data vs the world; the score that sails through
[1:07] Do this for itself, it has its training data. The supervisor has the world. When nobody audits, the failure is quiet. The score arrives in a credible format. Nobody asks whether it makes sense on its face and it goes through.
Two. Problem formulation: the question never asked
[1:20] Two, problem formulation. The chapter says most AI failures are failures of formulation, not performance. The triage system answered, "What statistical category is this presentation?" The clinical question was, "How likely is a life-threatening condition?" The first was answered well, the second was never asked. Three, tool orchestration. AI is
Three. Tool orchestration: AI, human, another model, or no tool yet
[1:39] One tool among many. For each subproblem, the supervisor picks the tool. The AI, human judgment, a different model, or the hardest call, no tool yet. We do not know how to do this well enough to deploy.
"Not here" is supervisory work; used because it is available
[1:50] Saying we should not deploy this here is supervisory work and often the most valuable work the supervisor does. The failure runs the other way. AI used because it is available, not because it is right for the task because nobody has the mandate to say no.
Four. Interpretive judgment: 5% at a spam filter vs a cancer screen
[2:05] Four, interpretive judgment. The chapter's example, a 5% false positive rate is fine for a spam filter and catastrophic for a cancer screen. The same number two meanings. The number is not the meaning. The meaning is the number in the domain. The failure, numbers read off a dashboard with no domain translation.
Five. Executive integration: where the buck stops
[2:22] Five, executive integration. Outputs from many tools, many models, and many people synthesized into one decision that someone stands behind. Somewhere in the organization, the buck has to stop. If there is no such place, the system has no decision-maker, only outputs.
The rarest capacity; decisions nobody owns
[2:38] The chapter calls this the rarest capacity and the one most often missing from AI deployments because no model produces it and most pipelines have no place for it. The failure, the pipeline produces decisions nobody owns, and when something goes wrong, accountability dissolves into the architecture.
Name the capacity at every step; the gap
[2:55] These five are vocabulary for now. By the end of the book, you can look at any deployment and name for each step which capacity is exercised and by whom. Where you cannot name it, you have found a gap, and in the chapter's words, gaps are where the patients die.
Your Turn
[3:08] Your turn, pick one AI deployment you know or one step of one and paste this into Claude. Here's one AI deployment I know or one step of it, describe it. For each of the five supervisory capacities, plausibility, auditing, problem formulation, tool orchestration, interpretive judgment, and executive integration, write who exercises it in one line each or write nobody. Then tell me which line you are least sure of and why. Then check two things yourself. Is every who a named role, not the team? And did you write nobody at least once honestly? Five things the model can't do for itself at Nick Bear Brown.
