Forcing the model to do arithmetic in Python

I built an internal Claude Code plugin at KnowBe4 that lets Claude answer business questions against a read-only context engine: account health, churn scores, subscription data, that kind of thing. Early on I noticed the model would read three numbers back from a tool call and then, in the same breath, add them, divide them, or compute a percentage change, entirely in prose. The arithmetic was often wrong. Not dramatically wrong, wrong in the specific way large language models are wrong at multi-digit math: confident, plausible, and off by an amount that a human skimming the answer would never catch.

The fix I did not build is a system prompt instruction telling the model to always use a calculator tool for math. I tried that first. It works most of the time, which for a compliance-adjacent answer about churn risk or revenue is not good enough. A model that forgets to use the calculator on one answer out of fifty still produced fifty answers that all read as equally confident. The person reading the output has no way to tell which one skipped the step.

Don't ask the model, check the transcript

The actual fix doesn't ask the model whether it did the math correctly. It doesn't ask the model anything. It's a Stop hook, also wired to SubagentStop so it fires under any persona subagent as well, that reads the raw transcript after the model has finished generating and mechanically checks three things: did a numeric field get read from a tool result earlier in the conversation, does the final answer text match a regex for prose arithmetic (numbers being summed, averaged, or turned into a percentage or ratio inline), and did an actual deterministic calculation tool, a calc.py or metrics.py style script, get invoked anywhere in between. If numeric data was read, the answer looks like arithmetic, and no calculation tool ran, the hook exits 2 and writes a block decision to stderr. The model has to redo the answer, this time routing the math through the tool.

None of this depends on the model's own account of what it did. It depends on whether a specific tool call is present in the transcript. That's the whole design principle: don't trust an LLM to self-certify its own arithmetic, verify it the same way you'd verify anything else a program claims to have done, by checking the log.

There's a cheap optimization underneath that check worth naming because it shapes how the hook behaves under load: a grep -F fast path runs first, on the order of a millisecond, to see whether the transcript mentions the read tool at all before paying for a full jq parse, which runs five to fifteen milliseconds. Most turns never touch numeric data, so most turns never pay for the expensive check.

Fail open on purpose

If the hook can't parse the transcript cleanly, for instance the JSON is malformed, or the transcript path isn't readable, it exits 0 and lets the response through rather than blocking silently. That's a deliberate tradeoff, not an oversight. A hook that fails closed on any parse error turns a formatting edge case into a hard stop on legitimate answers that have nothing to do with arithmetic. The cost of failing open is that a genuinely bad answer occasionally slips past the gate on a transcript the parser choked on. The cost of failing closed is that ordinary answers routinely get stuck behind a bug in the hook itself. I judged the second failure mode worse, because it's the kind of thing that erodes trust in the whole guardrail: if the hook blocks things it shouldn't, people start working around it.

Verifying the enforcement mechanism enforces anything

An enforcement mechanism is only as good as your evidence that it actually enforces. An earlier version of this hook piped the final-answer text into grep instead of using a herestring. The bug wasn't caught by testing this hook directly: a CI flake in an unrelated script's substring-match test led to an audit of every place in the plugin doing the same kind of check, and this hook's regex test turned out to be the one place a long enough answer really could get cut off, because the 64 KB pipe buffer combined with pipefail can let grep exit before it's seen the whole input. No confirmed production instance of the block silently failing to fire was ever found, but the audit fixed it anyway. The fix was switching to a herestring so the final answer text sits in memory for the check rather than streaming through a buffer that can get cut off mid-read. It's a boring bug, a buffer size, but it's the kind of thing an unrelated failure elsewhere can surface faster than staring at the code that actually has the bug.

What breaks the fast path

The grep -F check only stays cheap as long as it's looking for one thing: whether the transcript mentions the read tool at all. The day a second read-only tool starts exposing numbers worth gating on, that fast path either grows a second pattern or stops being a fast path, and nobody's forced to notice which one happened until someone benchmarks the hook again.