Build a Latency Predictor with Claude Code

"How fast is my model?" has four answers, and only their sum ships. This video builds a latency predictor with Claude Code that decomposes inference into pipeline stages — weight load, compute, memory movement, overhead — and estimates each stage's wall-clock on a target. The dominant stage for memory-bound models is almost never compute.

Built with Claude Code + Manim. Part of the Embedded AI with Claude series.

Nik Bear Brown · @NikBearBrown nikbearbrown.com

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

[0:00] Nick Bear Brown, build it with a CLI, then take it apart. How fast is my model? That question has four answers, and only their sum ships: weight, load, compute, memory, movement, overhead. Each one can dominate, and the one that dominates depends on the model's arithmetic intensity. Optimizing the stage that isn't the bottleneck is the most common wasted week in embedded ML. The predictor tells you which stage to fix before you spend the week. Ask for the experiment in the terminal claude write latency.pi given a model's max weight bytes activation bytes and a targets gflop s plus bandwidth plus overhead estimate per stage time and total. Read it before trusting it.

[0:44] Weight load is weight bytes divided by bandwidth. Compute is max divided by gflop ss. Memory movement is activation bytes divided by bandwidth. Overhead is a constant. Total is the sum. Which stage dominates depends on arithmetic intensity, max per bite. Memory bound models spend their time in weight load and memory movement, not in the multiplier. Run it. The four pipeline stages fill left to right like a gant chart. Weight load 45 milliseconds. The dominant stage in Crimson. This model is memory bound. Compute 12 milliseconds. Memory movement 38 milliseconds. Overhead 5 milliseconds, total 100 milliseconds. Weight load and memory movement together are 83% of the time. The compute stage isn't the problem. NT8

[1:34] Makes the model four times smaller. That shifts arithmetic intensity. Iterate clawed rerun for the NT8 model. Show which stage shrinks. Rerun. Int8 drops weight bytes by 4x. Weight load falls from 45 to 11 milliseconds. Memory movement drops from 38 to 10. Compute stays at 12. The max are unchanged. Only the bytes moved. Total 38 milliseconds. Now compute is the dominant stage in Crimson. The bottleneck shifted. A different knob moves it from here. So the bottleneck depends on arithmetic intensity, not total flop count. Memory bound weight load and memory movement dominate. Quantization moves. those computebound compute dominates pruning or a faster chip moves that optimizing the stage that isn't the bottleneck is

[2:26] The wasted week. Your move get your model's MAC count and weight bytes from a profiler and your targets G flops and memory bandwidth from the data sheet. Run the four- stage decomposition. The dominant stage tells you which lever to pull, not the total latency, not a benchmark, the stage breakdown. Nick Bear Brown. Build it with a CLI, then take it apart at Nick Bear Brown.