Distillation: See the Soft Label's Extra Signal

A one-hot label says "cat." The teacher says "85 % cat, 5 % dog, nothing else" — and that 5 % is the part that trains a better student. This video builds the distillation demo with Claude Code: teacher logits at temperature T = 1, 3, 6, the mixed loss α·hard + (1−α)·soft, and a bar chart showing the wrong-class signal that hard labels discard.

Built with Claude Code + Manim. Part of the Embedded AI with Claude series.

Nik Bear Brown · @NikBearBrown nikbearbrown.com

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

[0:00] Nick Bear Brown, build it with a CLI then take it apart. A one-hot label says cat. The teacher says 85% cat, 5% dog, nothing else. And that 5% is the part that trains a better student. The hard label discards everything except which box to tick. The teacher's soft probabilities carry the relative mistakes, how much the model confused cat with dog, but never with airplane. And that structure is exactly what a small student model most needs to learn from. Ask for the experiment. In the terminal, Claude, write distilled demo.py teacher logits for a 10-class cat image. Compare the one-hot label versus softmax of logits over T for T

[0:42] Equals 1, 3, and 6. Show the mixed loss and how temperature controls what transfers. Read it before trusting it. Temperature divides the logits before softmax. At T equals 1, you get the peaked distribution. At T equals 6, the distribution flattens and the wrong class probabilities become visible. The mixed loss alpha controls the trade-off. Hard label for accuracy, soft label for the teacher's dark knowledge. Run it. 10-class bars, cat in crimson on the left. At T equals 1, the cat bar towers and the nine wrong classes are nearly invisible. The one-hot label stands rigid and flat beside it. The soft distribution has signal in it. The hard

[1:25] Label has none. That's the T equals 1 picture. The signal is still hidden. Iterate, Claude. Pair it with pruning. Distill the pruned student against the float teacher. Show accuracy, Claude, back. Rerun. Two accuracy bars, hard label baseline on the left, distilled student on the right. The distilled student gains 3% over hard label training alone. The pruned network recovered the accuracy that pruning took away, not by adding parameters, but by training with the teacher's signal. The teacher's reasonable mistakes are the signal. Hard labels discard exactly the information a small model most needs, the relative distances between classes. Temperature exposes it. The mixed loss transfers it. That is why a distilled

[2:12] Student beats a student trained on the same data with hard labels alone. Your move. Run the temperature sweep on your own teacher's logits and watch where the wrong class probabilities become visible. The 2 to 5% accuracy gain is only real if you actually run distillation. That training run is yours. The demo today shows the mechanism. Your data set confirms the gain. Nick Bear Brown, build it with a CLI, then take it apart. @nickbearbrown nickbearbrown.com