Don't trust a model's arithmetic, only its tool calls

I built a calculator MCP server at KnowBe4. That sentence sounds like overkill until you look at what it's actually preventing. A language model asked to divide, threshold, or classify a business metric in the middle of a prose answer will usually get it right. Usually is the problem. Arithmetic error inside a paragraph of generated text has no error bar, no log line, and no way for a downstream consumer to tell a correct answer from a confidently wrong one. So I pulled every numeric operation out of the model's own reasoning and put it behind a tool call that runs actual code.

Why prose math fails quietly

The failure mode isn't the model being bad at arithmetic in isolation. It's what happens when arithmetic sits inside a longer chain of reasoning over real data: NULL fields that get silently coerced to zero, API responses that return numbers as strings and get concatenated instead of summed, thresholds that drifted in a config file the model never saw. None of that shows up as an exception. It shows up as a plausible-looking number that is wrong, embedded in a sentence that reads exactly like every other sentence the model writes. A tool call either returns a number or throws. Prose math just returns a number, and you find out later, if you find out at all.

The fix I landed on was a calculator and threshold-classification service, ported close to line for line from existing Python scripts, exposed as MCP tools. The model doesn't compute. It calls the tool, and the tool computes.

Forcing the tool call, not requesting it

Exposing a calculator tool isn't enough by itself, because a model can always choose to answer inline instead of calling it. I write a short preamble into every skill that touches computed numbers, something close to: this query involves computed values, use one of these deterministic executors before answering, in priority order, do not compute prose arithmetic on these fields yourself. It names the tool explicitly and orders the fallbacks. That preamble is shared text, referenced by every skill that's calc-gated rather than pasted into each one, so the instruction stays identical everywhere it applies instead of drifting skill by skill.

The point isn't politeness toward the model. It's that a numeric answer without a tool call behind it is not a valid answer for this class of query, and the preamble is the mechanism that makes that true before generation starts rather than something checked after the fact.

What the data actually looked like once I stopped assuming

The clearest argument for this pattern came out of an investigation into phish-prone percentage, a metric from the security-awareness training data this calculator runs over. Some fraction of the raw values came in above 100 percent, which on its face reads as a bug: a percentage over 100 look like an upstream double-count or a decimal-place error, and the obvious fix is to divide by 100 and move on.

I didn't take that shortcut. I pulled the actual distribution instead. Out of roughly 3.18 million raw rows with values over 100, 92.6 percent clustered between 100 and 200, not spread randomly the way a scaling bug would spread them. Then I looked at the specific values inside that cluster and found numbers like 133.33, which is exactly 4/3. That's not what corrupted data looks like. Corrupted data doesn't land on clean fractions. Real ratios do. The metric's numerator counts phishing events, not distinct users, so someone who fails several simulated-phishing tests can legitimately score above 100 percent. The values were correct. My assumption about what "over 100" meant was the thing that needed fixing.

Dividing those rows by 100, the fix I almost shipped, would have turned a real 133.33 percent into a plausible-looking 1.33 percent. That number would have passed every sanity check downstream, because it looks exactly like a normal, low, in-range phish-prone percentage. It would have been silently wrong in a way nothing would ever catch. The classifier now rejects out-of-range values as an explicit out-of-range finding and reports them as a data-quality question rather than rescaling them into something that merely looks reasonable.

A version number that meant nothing

The same calculator service reads a shared thresholds config that's meant to be byte-identical across three repositories. It wasn't. It had forked into three different actual contents, and two of those forks both declared themselves version 1.1.0, reached independently through two unrelated changes in two different repos. The version field was true in the sense that someone wrote it down, and false in the sense that it identified nothing about what was actually in the file. I made the loader refuse to run below a supported major.minor rather than trust the string. A version number that doesn't identify its content has stopped functioning as a version number. It's just a comment nobody is required to keep honest.

What the loader still can't tell you

Refusing to run below a supported major.minor stops the calculator from silently using thresholds it was never tested against. It doesn't tell you that a fourth copy of the config has drifted the same way the first two did, only that whichever one is running now is at least new enough to trust. That's a narrower guarantee than "the thresholds are correct," and it's the only one a version string can actually give you.