---
title: The judge that never fires
description: A quality check the model can choose to invoke isn't weak enforcement. It's no enforcement, and the corpus behind it has a measured number to prove it.
date: 2026-09-22
area: ai
tags:
  - claude-code
  - agents
  - patterns
draft: false
---

I built a Claude-run interview tool for the company I work for that replaces a manual intake process: someone
wants to propose a project, the tool asks plain questions in the submitter's own words, and it
quietly derives a structured, scored record behind the scenes without ever exposing the scoring
vocabulary to the person answering. The first version shipped with a validator subagent that
reviewed the whole draft before a human saw it. I called that validator a gate. It wasn't one.

## Advisory and gating nothing are the same sentence

The validator was a tool the orchestrating skill could call if it judged the draft needed a second
opinion. Across a batch of real interview sessions, that tool fired zero times, and two of those
sessions had answers that genuinely warranted a second look. Nothing malicious happened. The model
behaved reasonably in every one of them, it just kept doing what looked like the right thing in the
moment, which was moving forward, asking a clarifying follow-up, never once reaching for a check
that existed but that nothing obligated it to reach for.

That's the uncomfortable nuance in this failure mode. It isn't a story about a model cutting
corners. It's a story about a check whose only trigger was another model's judgment that the check
was needed. Judgment-gated judgment is a suggestion with better branding than the word suggestion
usually gets. A tool the client elects to call is a check nothing actually triggers, and a corpus
with a literal zero-firings count across real sessions is about as clean a proof of that as you'll
find.

## Moving the check inside the handler

The fix wasn't a better prompt telling the model to remember to validate. It was moving the check
out of the tool surface entirely and into the code path the client can't skip. In the redesign,
every recorded answer runs through a mechanical check first, unconditionally, before anything else
happens with it. Only if that mechanical pass is clean does a model-run judgment step even get
invoked, and even then it isn't offered to the model as an optional tool call. It's baked into the
handler that processes the answer. There is no code path that reaches "answer recorded" without
passing through the check first.

This has a corollary worth naming on its own, because it's easy to miss: removing the judge as a
callable tool also removes a budget-exhaustion attack surface. A tool a model can choose to invoke
is also a tool a model can invoke repeatedly, or invoke in a way that burns turns without
converging. Taking the judge off the tool surface didn't just fix the zero-firings problem. It
closed a second problem nobody had gone looking for yet.

## Where this generalizes

The specific bug was scoped to one interview tool, but the shape of it isn't specific to interview
tools at all. Any workflow with a "should" step, a validation pass, a compliance check, a
second-opinion review, has the same failure mode available to it the moment that step is exposed as
something an agent can decide to invoke rather than something the surrounding system forces it
through. It reads as reasonable architecture right up until you measure how often the optional path
actually gets taken under real use, and the honest answer is often close to never, for the same
reason a person rarely double-checks their own work unprompted: the work already feels done.

Tool-surface placement is a security decision, not just an architecture one. Where you put a check,
on the model's side of the boundary as a thing it can reach for, or on the system's side as a thing
it can't avoid, decides whether that check is real. I'd rather build the boring version, a
handler that always runs the mechanical pass before anything downstream sees the answer, than the
version that trusts a model to remember it has a validator available and reach for it on its own.
The boring version is the one with a nonzero firing count.