Build a Pareto Model-Selector with Claude Code

Benchmark accuracy lies to you. The leaderboard winner loses on your device — so plot the whole field and keep only the models nothing beats on every axis. This video builds the Pareto frontier tool with Claude Code: place N models on the accuracy-vs-latency plane, grey out dominated points one by one, then apply a 250 ms constraint mask to recompute the frontier.

Built with Claude Code + Manim. Part of the Embedded AI with Claude series.

Nik Bear Brown · @NikBearBrown nikbearbrown.com

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

[0:00] Nick Bear Brown, build it with a CLI, then take it apart. The benchmark accuracy lies to you. The leaderboard winner has 94% and tops every chart. Put it on your device, it runs in 400 milliseconds and misses the deadline. A model at 90% fits in 60 milliseconds. You would have picked it if you had plotted the whole field. The leaderboard optimizes for the leaderboard's constraints, not yours. The Pareto frontier is what model selection actually is. Ask for the experiment. In the terminal, Claude, write pareto.py given n models with accuracy, latency in milliseconds, and flash in kilobytes, compute the Pareto frontier and mark it, then gray out the dominated models. Read

[0:45] It before trusting it. A model is dominated if another candidate beats it on every axis simultaneously. Higher accuracy, lower latency, smaller flash. The frontier is the set no other model dominates. The leaderboard winner often falls off the frontier once latency and flash enter the comparison. Run it. Eight models appear on the accuracy versus latency plane. One by one, the dominated models transform to gray. F, G, and H each lose because another model beats them on every axis. The Pareto frontier draws itself through the survivors, C, B, A, D, and E. The leaderboard winner, model E at 94%, survives, but only because nothing else beats it on all three axes. The frontier

[1:35] With no constraints still includes models you can't use. Apply a deadline. Iterate. Claude, add a constraint mask, gray out anything over 250 milliseconds before computing the frontier. Rerun with a 250 millisecond cap. Model E at 400 milliseconds transforms to gray. Constraint violation. The new frontier recomputes from the remaining models, the leaderboard winner is now off the short list. The frontier you actually use is smaller and every model on it fits your deadline. A single number benchmark optimizes for the benchmarks constraints. The Pareto frontier optimizes for yours. Dominated models are eliminated before you spend time evaluating them. Constraint masks cut the frontier to what can actually ship.

[2:22] This is what model selection is. Your move. Collect the real accuracy, latency, and flash numbers for your candidate models. Accuracy from your eval set, latency and flash measured on the target hardware, and run pareto.py on them. The output beats in this video use illustrative numbers. Swap in your measurements to get the frontier that actually applies to your device. Nick Bear Brown. Build it with a CLI, then take it apart. At Nick Bear Brown, nickbearbrown.com.