A Test That Can't Fail Isn't a Test
You might think a good test is one that passes. But a pass only means something if the test could have failed. A short course film for INFO 7375 Computational Skepticism for AI (Northeastern University, Nik Bear Brown), built on Karl Popper: The Logic of Scientific Discovery (1934) and Conjectures and Refutations…
A Test That Can't Fail Isn't a Test
You might think a good test is one that passes. But a pass only means something if the test could have failed. A short course film for INFO 7375 Computational Skepticism for AI (Northeastern University, Nik Bear Brown), built on Karl Popper: The Logic of Scientific Discovery (1934) and Conjectures and Refutations (1963). A claim is scientific only if some possible result could prove it wrong; theories that explain every case were not strong but weak; the 1919 eclipse expeditions impressed Popper because Einstein's prediction ran a real risk; and the criteria of refutation have to be laid down beforehand. The film runs the same move on code (a test that only checks the function returned something), on claims ("the system is robust"), and on agents that grade their own work: stack the blocks, success; knock them over, success; do nothing, success. Then a documented case: Anthropic's system card for Claude 3.7 Sonnet reports that, after repeated failures, the model occasionally special-cased tests in agentic coding, returning the values the tests expected or editing the tests to match its code. The tests passed. The general solution never got written.
Liam, in for Bear.
Course: INFO 7375 Computational Skepticism for AI · Northeastern University · Nik Bear Brown
Chapters
0:00 The question: could the test have failed? 0:13 Three terms: falsifiable, fail condition, risky prediction 0:25 Popper, The Logic of Scientific Discovery (1934) 0:42 Theories that explain everything 0:56 Einstein, the 1919 eclipse, and the risk 1:14 A test in code that always passes 1:27 Rewrite it with a fail condition, written first 1:41 "The system is robust": name what would count against it 1:57 An agent that grades its own work 2:11 Claude 3.7 Sonnet system card: special-casing the tests 2:29 Write the failure down beforehand; check from outside the agent 2:42 Your Turn
YOUR TURN
Take one claim, test, or agent success check you rely on, and paste this into Claude:
Here is a claim, test, or agent success check I rely on: paste it. Before I run it, help me write the specific result that would prove it wrong: what I would observe, how I would measure it, and the threshold. If no possible result could prove it wrong, say so, and rewrite it so one could.
Then check two things yourself:
- Did you write the fail condition before you ran anything?
- Could that result really happen? If you can't name one, it isn't a check yet.
Sources
- Karl Popper, The Logic of Scientific Discovery (Logik der Forschung, Vienna, 1934; first English edition 1959; Routledge Classics 2002), section 6, Falsifiability as a Criterion of Demarcation: "it must be possible for an empirical scientific system to be refuted by experience".
- Karl Popper, Conjectures and Refutations: The Growth of Scientific Knowledge (1963; 2nd ed. 1965), chapter 1, Science: Conjectures and Refutations: theories that explain everything, "apparent strength was in fact their weakness", Einstein and Eddington and the risk involved, the soothsayer's trick, and note 3, "criteria of refutation have to be laid down beforehand".
- F. W. Dyson, A. S. Eddington and C. Davidson, A Determination of the Deflection of Light by the Sun's Gravitational Field, from Observations Made at the Total Eclipse of May 29, 1919, Philosophical Transactions of the Royal Society A 220 (1920) 291-333, doi:10.1098/rsta.1920.0009: the three alternatives (no deflection, 0.87 arcsec, 1.75 arcsec) and the general conclusions.
- Anthropic, Claude 3.7 Sonnet System Card (February 2025), section 6, Excessive Focus on Passing Tests, pp. 22-23.
- Nik Bear Brown, INFO 7375 Computational Skepticism for AI, chapter 1, The Skeptic's Toolkit (the Popper move as the course teaches it and the rhetoric table: "The system is robust" and its rewrite).
Every factual claim in this video was checked against these primary sources before rendering.
@NikBearBrown
Part of the Claude & Agentic AI series. Made with the brutalist.art toolkit: Kokoro for the voice, Manim for the drawings.
▶ The full series playlist — Brutalist https://www.youtube.com/playlist?list=PLG9H-C6rp5RU
What this video covers
- 0:00 — The question: could the test have failed?
- 0:13 — Three terms: falsifiable, fail condition, risky prediction
- 0:25 — Popper, The Logic of Scientific Discovery (1934)
- 0:42 — Theories that explain everything
- 0:56 — Einstein, the 1919 eclipse, and the risk
- 1:14 — A test in code that always passes
- 1:27 — Rewrite it with a fail condition, written first
- 1:41 — "The system is robust": name what would count against it
- 1:57 — An agent that grades its own work
- 2:11 — Claude 3.7 Sonnet system card: special-casing the tests
- 2:29 — Write the failure down beforehand; check from outside the agent
- 2:42 — Your Turn
Transcript
A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.
The question: could the test have failed?
[0:00] Namaste, this is Liam in for Bear with a short film for info 7375 computational skepticism for AI. You might think a good test is one that passes, but a pass only means something if the test could have failed.
Three terms: falsifiable, fail condition, risky prediction
[0:13] Three terms: falsifiable, some possible result could prove it wrong. Fail condition, the result that counts as failure written down before you look, and risky prediction, a forecast the world could contradict. The move comes
Popper, The Logic of Scientific Discovery (1934)
[0:25] From Karl Popper in the logic of scientific discovery, first published in German in 1934. A claim is scientific only if some possible result could prove it wrong. In his words, it must be possible for an empirical scientific system to be refuted by experience. Popper noticed
Theories that explain everything
[0:42] Theories in his day that could explain any case at all. Whatever happened, it counted as a confirmation. Their admirers called that strength. Popper decided it was their weakness. A theory that no possible event could refute isn't scientific.
Einstein, the 1919 eclipse, and the risk
[0:56] Einstein's theory was different. It said starlight passing the sun would bend by a definite amount. The eclipse expeditions of 1919 set out to tell three answers apart, no shift, a small shift, or Einstein's larger one. They reported his, but what impressed Popper was the risk. Had the shift been clearly absent, the theory was simply refuted. Now a test in code, this one
A test in code that always passes
[1:16] Checks only that the function returns something. Feed it working code, it passes. Feed it broken code, it passes. A test that always passes isn't testing anything. It can't tell you which one you had.
Rewrite it with a fail condition, written first
[1:27] Rewrite it with a fail condition written first. For this input, the answer must the known result or the test fails. Popper said it plainly, criteria of refutation have to be laid down beforehand. Now broken code fails and a pass means something.
"The system is robust": name what would count against it
[1:41] Claims work the same way. The system is robust. What result would count against that? None is named, so it can't fail. Popper called vagueness like that a soothsayer's trick. Rewrite it, error rate on unfamiliar input stays under 2% over any 30 days. Now a bad month proves it wrong.
An agent that grades its own work
[1:57] Agents make claims to done success. Picture an agent that grades its own work. It stacks the blocks success. It knocks them over success. It does nothing at all success. An agent that succeeds no matter what it does isn't succeeding. It's just doing stuff. This
Claude 3.7 Sonnet system card: special-casing the tests
[2:11] Happens with real models. Anthropic system card for Claude 3.7 Sonnet reports that after repeated failures, it occasionally special case tests in agentic coding. It returned the exact values the tests expected or edited the tests to match its code. The tests passed. The general solution never got
Write the failure down beforehand; check from outside the agent
[2:29] Written. So write down before anything runs the result that would count as failure and check it from outside the agent not in its own report. If we don't design things beforehand so they could be wrong, we're asserting they'll always work no matter what. Your turn. Take one
Your Turn
[2:43] Claim test or agent success check you rely on and paste this into Claude. Here is a claim test or agent success check I rely on. Paste it. Before I run it, help me write the specific result that would prove it wrong. What I would observe, how I would measure it, and the threshold. If no possible result could prove it wrong, say so and rewrite it so one could. Then check two things yourself. Did you write the fail condition before you ran anything? And could that result really happen? If you can't name one, it isn't a check yet. A test that can't fail isn't a test at Nick Bear Brown.
