Deploy a Keyword Spotter: Find the Layer That Leaked Accuracy

The model was 94 % accurate in Python. After the toolchain got done with it, it was 91 % — and the 3 % vanished in a step nobody watched. This video builds the deployment runner with Claude Code: TFLite int8 conversion, interpreter simulation on a val set, and a layer-by-layer bisection to find which op introduced the largest output divergence.

Built with Claude Code + Manim. Part of the Embedded AI with Claude series.

Nik Bear Brown · @NikBearBrown nikbearbrown.com

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

[0:00] Nick Bear Brown build it with a CLI then take it apart. The model was 94% accurate in Python. After the tool chain got done with it, it was 91% and nobody could say where the 3% went. Quantization is the obvious suspect, but tool chain induced accuracy loss is a separate failure. An unsupported opusion, a range mismatch, a CMSN fallback that silently changes the output. The only defense is a diff at every hop. Bisect which layer introduced the divergence before you blame the wrong thing. Ask for the experiment in the terminal claude. Write deploy runner. py convert a caris model to tflight n8. Run the tflight interpreter

[0:44] On a validation set. Compare it to the float model and biseect which layer or op introduce the largest output divergence. Read it before trusting it. The bisection walks the network layer by layer and compares the float model's intermediate output against the TF light interpreter's output at the same point. Divergence is mean squared error between the two. A small divergence is normal quantization noise. A spike is a tool chain failure, an unsupported fusion or a kernel that rounds differently than the float version. Run it. 10 layers plotted left to right. Divergence on the vertical axis. Layers 0 through six. Small bars. Normal quantization noise. Layer seven, a crimson spike. MSE jumps

[1:28] By two orders of magnitude. The bisection localizes the leak. This is the layer the tool chain broke, not quantization in general. Layer 7 is the spike. Now find out why. Iterate clawed. Add a per op supported and unsupported check for CMSisn and flag any fallbacks. Rerun with op support check. Above each layer bar, a chip appears. Okay. for layers with CMSSN kernel support. At layer 7, fall back. The op at that layer, a fused activation that CMSN doesn't support, falls back to a reference kernel that rounds differently than the float model. That is where the 3% went. The six-step deploy pipeline breaks silently between steps.

[2:14] Quantization error is one thing. Tool chain error is another. The bisection separates them by walking the network and measuring the divergence at each hop. The CMSisn OP check explains the mechanism. Fix the op rebuild and the 3% comes back. Your move. Bring a trained keyword spotter model in curis or TF light format and a validation set. Claude can't train these, but given them, it does the conversion, bisection, and opche in pure Python. No hardware required. The output beats in this video use illustrative divergence values. Run deploy runner.pi on your real model to get the honest layerbylayer diff. Nick Bear Brown. Build it with a CLI then

[2:58] Take it apart at nickbearbr. nickbear brown.com