A green test suite is not proof a non-programmer's edit was safe

I built an AI-assisted QA plugin at KnowBe4 that reviews e-learning course packages before they ship, replacing a chunk of a manual checklist a Publishing QA team ran by hand. The team that owns it day to day is not an engineering team. They are content QA specialists and content creators, and that fact was true on day one and stayed true. I wrote it into an early architecture decision record for the project: none of the people maintaining this tool are programmers, and I treated that as the permanent shape of the team rather than a staffing gap to fix later. It became an input to the architecture instead of a caveat documented after the fact.

Two ways a maintained tool dies

A tool maintained by non-programmers can fail in two directions, and they look nothing alike. The first is ossification: nobody touches it because everyone is afraid to, so a wordlist that should gain three entries a week gains three a year, and the tool slowly stops matching the content it reviews. The second is silent degradation: someone reworks the AI rubric with good intentions, softens a phrase, drops a qualifier that mattered, and every one of the automated checks still passes because none of them were testing what changed. The first failure is slow and visible. The second is fast and invisible, and it is the one that actually breaks a QA tool, because a QA tool that quietly stops catching things is worse than no tool at all.

Designing against ossification alone would have been easy: write good docs, add a wiki page, wish people luck. Designing against silent degradation required something a document cannot provide, which is a way to tell, mechanically, whether a given edit is the kind that can be reasoned about by someone without a programming background, or the kind that needs a person who understands what the code is actually asserting.

A tier system that is also the enforcement

The answer was a three-tier change boundary, green, amber, red, and the deliberate decision to make it exist in two places that cannot drift apart: a table in the maintainer-facing runbooks, and a TIERS mapping inside the verification script itself. Green covers the spellcheck wordlist, the runbooks, the README, other docs. Change it yourself. Amber covers the AI rubric, keyframe selection, the check scripts, the report builder, the guardrail hook, the tests, the ADRs. Change it with a reviewer who understands the check. Red covers the core scripts, the fixtures the tests assert against, dependency pinning, CI configuration, the verifier itself, the bundled dictionaries. That needs an engineer and a recorded decision. Anything the map does not recognize defaults to red on purpose: an unclassified path stays treated as risky until someone actually classifies it.

The tier table does double duty. It is the policy document a maintainer reads, and it is also the exact data the verification script consults before it will call a change safe to propose.

The check that fails a green test suite

The single most important behavior in the whole system is the verifier's willingness to say safe: false on a change that passes every automated test. If a change touches a red-tier file, the verifier reports it unsafe regardless of how many of the 197 checks are green, because in that case the checks stopped being evidence about the thing that actually changed. A test suite proves the plumbing still runs. It does not prove that a rubric edit didn't quietly narrow what a check looks for, and treating a clean test run as sufficient permission for any edit is exactly the gap that lets silent degradation through. The verifier's job is to catch the class of problem a passing test suite is structurally unable to see, not to duplicate what the tests already cover.

Guided skills that refuse to skip steps

The interface for a maintainer is a set of guided Claude Code skills, not raw git. One of them will not open a merge request for a rubric change without a manual before-and-after comparison, and the tooling enforces that rather than asking nicely for it in a runbook. Every symptom-indexed runbook follows the same shape: what you saw, the one file that owns it and its tier, a table separating "the tool is wrong" from "the tool is right and the fix is elsewhere," the exact prompt to give Claude, and the one command to run afterward to confirm it worked. Claude drives the git mechanics and opens a draft merge request. The maintainer reads a diff and a verdict, not a terminal.

What moving the pipeline didn't change

A few weeks later, the deterministic parts of the pipeline moved to a hosted service reached over MCP, mostly to get out from under a laptop-side credential problem nobody had been able to solve, and to stop losing time to install failures. The tier system moved with it unchanged, because the problem it solves was never about where the code ran. A maintainer with no programming background still needs a way to know, before proposing an edit, whether that edit is the kind they can reason about alone. That question doesn't get easier by changing infrastructure. It gets answered once, correctly, by deciding who will maintain the system before writing the first line of the checker.