---
title: Forcing the model to do arithmetic in Python
description: An LLM cannot be trusted to self-report whether it actually did math, so a Stop hook checks mechanically whether a real calculation tool ran first.
date: 2026-09-22
area: ai
tags:
  - claude-code
  - agents
  - patterns
draft: false
---

I built an internal Claude Code plugin at the company I work for that lets Claude answer business questions
against a read-only context engine: account health, churn scores, subscription data, that kind of
thing. Early on I noticed the model would read three numbers back from a tool call and then, in the
same breath, add them, divide them, or compute a percentage change, entirely in prose. The
arithmetic was often wrong. Not dramatically wrong, wrong in the specific way large language models
are wrong at multi-digit math: confident, plausible, and off by an amount that a human skimming the
answer would never catch.

The fix I did not build is a system prompt instruction telling the model to always use a calculator
tool for math. I tried that first. It works most of the time, which for a compliance-adjacent
answer about churn risk or revenue is not good enough. A model that forgets to use the calculator on
one answer out of fifty still produced fifty answers that all read as equally confident. The person
reading the output has no way to tell which one skipped the step.

## Don't ask the model, check the transcript

The actual fix doesn't ask the model whether it did the math correctly. It doesn't ask the model
anything. It's a Stop hook, also wired to SubagentStop so it fires under any persona subagent as
well, that reads the raw transcript after the model has finished generating and mechanically checks
three things: did a numeric field get read from a tool result earlier in the conversation, does the
final answer text match a regex for prose arithmetic (numbers being summed, averaged, or turned
into a percentage or ratio inline), and did an actual deterministic calculation tool, a `calc.py` or
`metrics.py` style script, get invoked anywhere in between. If numeric data was read, the answer
looks like arithmetic, and no calculation tool ran, the hook exits 2 and writes a block decision to
stderr. The model has to redo the answer, this time routing the math through the tool.

None of this depends on the model's own account of what it did. It depends on whether a specific
tool call is present in the transcript. That's the whole design principle: don't trust an LLM to
self-certify its own arithmetic, verify it the same way you'd verify anything else a program claims
to have done, by checking the log.

There's a cheap optimization underneath that check worth naming because it shapes how the hook
behaves under load: a `grep -F` fast path runs first, on the order of a millisecond, to see whether
the transcript mentions the read tool at all before paying for a full `jq` parse, which runs five to
fifteen milliseconds. Most turns never touch numeric data, so most turns never pay for the expensive
check.

## Fail open on purpose

If the hook can't parse the transcript cleanly, for instance the JSON is malformed, or the
transcript path isn't readable, it exits 0 and lets the response through rather than blocking
silently. That's a deliberate tradeoff, not an oversight. A hook that fails closed on any parse
error turns a formatting edge case into a hard stop on legitimate answers that have nothing to do
with arithmetic. The cost of failing open is that a genuinely bad answer occasionally slips past the
gate on a transcript the parser choked on. The cost of failing closed is that ordinary answers
routinely get stuck behind a bug in the hook itself. I judged the second failure mode worse, because
it's the kind of thing that erodes trust in the whole guardrail: if the hook blocks things it
shouldn't, people start working around it.

## Verifying the enforcement mechanism enforces anything

An enforcement mechanism is only as good as your evidence that it actually enforces. An earlier
version of this hook piped the final-answer text into grep instead of using a herestring. The bug
wasn't caught by testing this hook directly: a CI flake in an unrelated script's substring-match
test led to an audit of every place in the plugin doing the same kind of check, and this hook's
regex test turned out to be the one place a long enough answer really could get cut off, because
the 64 KB pipe buffer combined with `pipefail` can let grep exit before it's seen the whole input.
No confirmed production instance of the block silently failing to fire was ever found, but the
audit fixed it anyway. The fix was switching to a herestring so the final answer text sits in
memory for the check rather than streaming through a buffer that can get cut off mid-read. It's a
boring bug, a buffer size, but it's the kind of thing an unrelated failure elsewhere can surface
faster than staring at the code that actually has the bug.

## What breaks the fast path

The `grep -F` check only stays cheap as long as it's looking for one thing: whether the transcript
mentions the read tool at all. The day a second read-only tool starts exposing numbers worth gating
on, that fast path either grows a second pattern or stops being a fast path, and nobody's forced to
notice which one happened until someone benchmarks the hook again.