What The Economist Left Out

Jambo. This is Liam, in for Bear. Last time we read a study that says AI use costs Chinese schoolchildren twenty percent of their exam scores. This time we read the summary that took it to a million people — and then we go after the study itself. The charts were faithful. The last paragraph was not.

0:00 Introduction 5:03 Verdict 5:23 Your Turn

YOUR TURN: Find a news summary of a study you already believe, get the actual paper, and run this. The interesting output is never the errors — good outlets rarely get numbers wrong. It is the verbs. Linked to, associated with, boosts. Somebody chose those. Go see who.

Built with Kokoro am_onyx voice · Claude fidelity skin · Manim + Remotion visuals. Liam, in for Bear.

youtube.com/@NikBearBrown

What this video covers

• 0:00 — Introduction
• 5:03 — Verdict
• 5:23 — Your Turn

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

Introduction

[0:00] Jambo, this is Liam in for Bear. Last time we read a study that says AI use cost Chinese school children 20% of their exam scores. This time we read the summary that took it to a million people, and then we go after the study itself. The charts were faithful. The last paragraph was not. 18th of August, 2026. Graphic detail, 2 minutes long, three charts, a clean summary of a 58-page working paper. This is not a hit piece on The Economist. The charts are accurate, and short-form science writing is genuinely hard. But a summary is a set of choices, and the choices are the interesting part. Six claims side by side. The first two match exactly. 18% up on homework, 20% down on exams. Those are the paper's numbers, correctly reported. It is the other four that do the work. Here's the big one. The write-up ends, AI can boost learning productivity, but only for those who use

The technology intelligently. Go to the paper. Section 5.1. Students who use AI and spend the same time on homework as non-users achieve, quote, similar exam scores. Similar. The same. That is an absence of harm. The summary converted it into a presence of benefit, and put it in the last sentence, which is the sentence people remember. Second one, why do the careful students do fine? The write-up says, more likely they used the chatbots as a personal tutor. The paper lists four candidates: intrinsic motivation, parental monitoring, better information about what outsourcing costs, or simply more skill at using the thing to learn. It argues motivation is the least likely, and then declines to pick. The summary picked. That is a hypothesis wearing a findings clothes. And the omission. The paper's most decision-relevant finding, the one a parent or a head teacher would actually act on, is that the damage to the exams that matter keeps compounding for 2 years.

It is the reason the authors say short studies systematically understate the cost. It does not appear in the write-up at all. Right, our turn to be the unpleasant one. Everything you have seen rests on a single variable, the month each student says they first used AI. Not a log, not an app record the researchers pulled. One question asked once in June 2025 covering up to 30 months of recall, teachers told students to check their registration dates. Section 4.3, we cannot observe whether they actually did. Here's why that matters more than it sounds. The paper checks two kinds of error, reporting too late which would show up as a pre-trend and reporting too early which would slow the ramp. Both assume the mistake is random, but ask a 15-year-old to date a fuzzy event and they reach for a landmark. The semester it went wrong is a landmark. If even a minority anchor there, you manufacture flat pre-trends

And a sharp post-adoption drop of an ordinary bad year. To be fair, and this is the paper's real defense, recall bias should not produce three of these things. It should not reproduce the same subject ordering across two exam systems with different writers, graders, and stakes. It should not produce a clean monotone dose response. And it should not survive in the county-wide exams where every school sits the same paper and no teacher can quietly make it easier. Those are good arguments. They establish that something real is here. They do not establish that it is this big. The size, 1.4 standard deviations on the monthly average, 1.3 to 1.5 on the entrance exams. The paper prints this honestly. Averaging nine subjects compresses the standard deviation, so benchmark the usual way it is about half that. Call it 0.7. A full year of secondary school is worth a quarter to 4/10 of a standard deviation. So the claim is a

Chatbot costs you 2 years of school in 6 months. That might be true. It should be said out loud, and the paper never says it out loud. And one more smaller but real. The outsourcing story turns on a 50-minute cutoff. Where did 50 come from? This graph. It was read off the data after the fact along with the 45-minute line, the 6-to-10-month window, and the 24-month window. Each choice is defensible. Nothing was pre-registered. That is not fraud. It is degrees of freedom, and the paper never counts them. So who disagrees? Bastani and colleagues in PNAS. Turkish high schoolers, a chatbot with no guardrails, practice scores up, closed book scores down 17%. That agrees. Kesten and colleagues Harvard physics students a purpose-built AI tutor and it beats active learning in class. That disagrees. Contractor and raise 200 undergraduates AI assisted learning plus 0.27 standard deviations. And most of it's still there a week later. That disagrees, too. Except

They don't actually disagree and this is the reading neither headline gives you. Sort those four studies by tool and setting instead of by result and the split disappears. Design tutor supervised room motivated student helps. General purpose answer machine unsupervised bedroom weekend homework 50 kids in the class it hurts. The Economist pairs two of these and says they point one way. They point at two different questions. So here is the narrowest sentence the data will actually carry. In one Chinese county students who report adopting AI show flat pre-trends and then an 18% rise in homework marks, 19 fewer minutes per assignment, and a 20% fall in closed-book exams within 6 months concentrated among those whose homework time collapses. The entrance exam effects point the same way on a weaker design. And range from 5 to 24% depending on which specification you pick. That is still a big important publishable result. It is not 1 and 1/2 standard deviations.

Verdict

[5:03] The verdict on both documents, the paper a serious careful unusually honest piece of work whose headline number is further out on a limb than its abstract admits. The write-up three faithful charts and one causal upgrade that the paper explicitly disclaims. If you take one habit from this, take the boring one before you repeat a number. Find out whether the treatment was measured or remembered. Your turn, find a new

Your Turn

[5:24] Summary of a study you already believe, get the actual paper and run this. The interesting output is never the errors. Good outlets rarely get numbers wrong. It is the verbs linked to associated with boosts. Somebody chose those. Go see who. What The Economist left out, the paper, the write-up and the full skeptical read are linked below. Check my work. This is Liam For Bear.