Solving Is Not Verifying: Why AI's Biggest Gap Gets Wider as Models Get Better

Solving a problem and verifying a solution are two different cognitive operations. AI automates the first. The second stays human — and as the models get faster and more fluent, the gap does not close. It deepens.

This film defends that claim on its own terms, and marks exactly where it stops being airtight.

WHAT'S IN IT

• The misconception: "verifying is just solving again, more carefully." It isn't — verification is a different operation, needing inputs a solver structurally lacks.
• The computer-science analogy (Cook 1971, Levin independently; P versus NP) — and why it only gives permission, not proof: it's an open Millennium Prize conjecture, and language models aren't literally subject to worst-case complexity.
• The argument that actually holds for AI: a model generates by sampling a learned distribution, so a fabricated citation that looks like a real one is a likely continuation. Ask the same model to verify, and it samples the same distribution again — the loop closes on its own blind spot. Stechly, Valmeekam and Kambhampati tested this directly on GPT-4.
• What verification actually requires: grounding in specific reality. Does this case exist in the corpus of law? Does this package exist in the registry?
• Why scaling deepens the asymmetry: a wider SOLVE channel pours more finished-looking output at a VERIFY gate that stays human-sized.
• The honest counter-case: since 2024, reasoning models and tool-using systems verify more and better. But when a model checks a citation against a real legal database, the grounding lives in the tool, not the model. Tool use relocates the asymmetry; it doesn't refute it.
• The lawyer's room, re-examined step by step — and the two stories the asymmetry dissolves: the comfort ("the next model will check its own work") and the threat ("you'll just be a rubber stamp").

THE LINE TO KEEP Solving is not verifying. As the solving gets faster, the operation that grounds it does not — and the enterprise depends on the operation that cannot be sped up.

CHAPTERS 0:00 The boldest claim 0:19 The whole idea in one sentence 0:34 Act I — Two Operations, Not One Harder One 1:10 Act II — The Computer-Science Analogy, and Its Limits 2:04 Act III — The Argument That Holds for AI 3:24 Act IV — Why the Gap Deepens Instead of Closing 4:00 Act V — The Live Counter-Case: Reasoning Models and Tools 4:52 Act VI — The Worked Example: The Lawyer's Room 5:26 Act VII — Two Stories the Asymmetry Dissolves 6:08 The line to take with you 6:19 Verdict 6:48 Your Turn

YOUR TURN — paste this into Claude: Take one AI-assisted task from this week. Mark precisely where solving ended and verification began. Then name what the verification required that the solver could not supply — the specific reality the answer had to be grounded against. If you cannot name it, the verification did not happen.

SOURCES Cook (1971) and Levin, NP-completeness and P versus NP (Clay Millennium Prize problem) · Kambhampati on LLMs as approximate retrieval (position marked contested in the film) · Stechly, Valmeekam & Kambhampati, empirical tests of LLM self-critique on Game of 24, graph coloring and planning · reporting on fabricated legal citations in AI-assisted filings. Every claim was checked against its source before rendering.

AI DISCLOSURE Narration is a synthetic voice (Liam, in for Bear). Graphics are rendered, not stock.

Series: Computational Skepticism · Conducting AI @NikBearBrown

What this video covers

• 0:00 — The boldest claim
• 0:19 — The whole idea in one sentence
• 0:34 — Act I — Two Operations, Not One Harder One
• 1:10 — Act II — The Computer-Science Analogy, and Its Limits
• 2:04 — Act III — The Argument That Holds for AI
• 3:24 — Act IV — Why the Gap Deepens Instead of Closing
• 4:00 — Act V — The Live Counter-Case: Reasoning Models and Tools
• 4:52 — Act VI — The Worked Example: The Lawyer's Room
• 5:26 — Act VII — Two Stories the Asymmetry Dissolves
• 6:08 — The line to take with you
• 6:19 — Verdict
• 6:48 — Your Turn

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

The boldest claim

[0:00] Bonjour, this is Liam in for Bear. Here is the boldest claim I will defend today. Solving a problem and verifying a solution are different cognitive operations. AI automates the first. The second stays human, and as the tools get faster, the gap deepens rather than closes. In the next few minutes, I will show you exactly where the argument is airtight and where it stops being.

The whole idea in one sentence

[0:19] Here is the whole idea in one sentence. The next model will not solve verification because verification is a different operation altogether, one that needs grounding a solver does not contain. Watch the misconception correct itself. Act one, two operations, not one harder

Act I — Two Operations, Not One Harder One

[0:36] One. Here is the misconception this film exists to detonate. Ask most people whether AI will eventually verify its own work and they reason like this. Verifying is just solving again more carefully. If the model can solve, then with enough scale, it will solve the checking problem, too. Verification, they conclude, is a harder instance of generation. That is wrong and not because verification is harder. It is wrong because verification is a different kind of operation, not a more careful solve, a different operation requiring inputs the solver structurally does not have. Two operations, not one harder one.

Act II — The Computer-Science Analogy, and Its Limits

[1:10] Act two, an analogy from computer science and its limits. In 1971, Stephen Cook, with Leonid Levin working independently, gave the world the concept of NP-completeness and the question we now call P versus NP. Strip it to its core. There's a class of problems for which a candidate answer can be checked quickly. The open question is whether every such problem can also be solved quickly. The widespread belief, unproven, is that the answer is no. Finding is genuinely harder than checking. Now the honesty. The analogy is bounded twice over. First, P versus NP is an open conjecture, not a theorem. It is a clay millennium prize problem for exactly that reason. Second, and more important, large language models are not literally subject to it. P versus NP is about worst-case algorithmic complexity. A model does something better described as approximate retrieval from a learned distribution. Subbarao Kambhampati puts the caution plainly, treat his position as contested.

Act III — The Argument That Holds for AI

[2:04] Act three, the argument that holds for AI made on its own terms. Forget complexity classes, reason about what a model actually does. When a model generates it samples from a learned distribution, it produces the continuation that is in the statistical landscape of its training common and likely. That is why fabricated case citations look exactly like real ones. A citation that fits the form and frequency of real citations is a likely continuation whether or not the case exists. Now ask that same model to verify its answer. What does it do? It samples again from the same distribution that produced the answer in the first place. Self-verification is more retrieval against the same landscape that contained

The error. If the original mistake was a likely-looking fabrication, the verification pass finds it likely-looking, too. The retrieval loop closes on its own blind spot. This is not armchair reasoning. Steckley, Val Mican, Kambhampati tested it directly on GPT-4, game of 24, graph coloring, planning, and found this. Here is the load-bearing step. What verification actually requires in the lawyers' room, the physicians, the analysts, is grounding in the specific domain reality. Does this case exist in the actual corpus of law? Does this package exist in the actual registry? Do these sources actually exist? That grounding is exactly what statistical generation does not contain. It is an appeal to something outside the distribution. Act four, why the gap

Act IV — Why the Gap Deepens Instead of Closing

[3:25] Deepens instead of closing. Grant the argument for a moment, watch what scaling does. A faster, cheaper, more fluent model widens the solve channel every year, broad, fast, optimized for the common and the likely. The verified gate is narrow, situated, accountable, human-sized. More water through the wide channel does not widen the gate, it just means more is arriving at it faster, looking more finished than ever. That is the structural prediction, and it is the opposite of the comfortable one. The asymmetry deepens not because the models are weak but because the operation the verification requires was never in the machine. Act five, the live counter case honesty

Act V — The Live Counter-Case: Reasoning Models and Tools

[4:01] Demands. Since 2024 reasoning models, systems that spend inference time compute on extended intermediate steps, and frontier systems that verify against external tools have plainly verified more and verified better than the models the original critique was written against. Any honest treatment concedes it. The gap is narrower in 2026 than the simplest version of the argument implies. Then notice where the grounding actually comes from. When a reasoning model verifies a citation by querying a real legal database, the verification is grounded not by the model but by the tool. When it runs code to confirm a result, the grounding lives in the executable specification a human wrote. The model is increasingly good at orchestrating grounded checks. It is not the source of the grounding. So the claim survives suitably bounded. Tool use does not refute the asymmetry, it relocates it. Act six, the worked example applied with

Act VI — The Worked Example: The Lawyer's Room

[4:53] Precision. Return to the lawyer's room. Ask, was the AI's output solved correctly? Walk the re-examination step by step. And it was catastrophically wrong because the operation the situation required was not a better solve. It was a different operation entirely. Do these cases exist in the actual corpus of law? That question cannot be answered from the distribution of citation shaped text. Confronted, the attorney re-prompted asking the solver to solve the checking question and the solver solved it the only way it could. It produced more likely looking text and it confirmed the fabrication. Act seven,

Act VII — Two Stories the Asymmetry Dissolves

[5:26] Two stories the asymmetry dissolves for the same reason. The comfort relaxed, the next model will check its own work. This is the misconception dressed for the future. A better solver produces better looking output that needs the same external verification, arguably more urgently because it is harder to spot as wrong. The comfort is not just false, it points the wrong way. The threat verification is the leftover diminished work. You will be a rubber stamp. This inverts what verification is. The solver faces no court, no patient, no refund. The verifier signs. Far from being the residue of the process, verification is the part with the consequence in it. The faster the solving, the more the whole enterprise depends on the one operation that cannot be sped up by the same means. Take this

The line to take with you

[6:08] Line with you. Solving is not verifying. As the solving gets faster, the operation that grounds it does not. And the enterprise depends on the operation that cannot be sped up.

Verdict

[6:19] Verdict, the boldest claim defended. Solving and verifying are different cognitive operations. AI automates the first. Verification requires grounding, an appeal to specific reality that statistical solver does not contain. Cook 1971 gives us permission, not proof. Self-verification is more retrieval against the same blind spot. Reasoning models relocate the verification into the tools they call. They do not replace the human who chooses the tool and signs. As solving scales, the asymmetry deepens. Your turn. Open a Claude session. Paste

Your Turn

[6:49] This prompt. Take one AI-assisted task from this week. Mark precisely where solving ended and verification began. Then name what the verification required that the solver could not supply, the specific reality the answer had to be grounded against. If you cannot name it, the verification did not happen. That paragraph is this idea lived. The solve-verify asymmetry lay in forbear.