A green pipeline is not evidence

Two gates in one of my repos existed specifically to catch a version-bump and changelog mismatch before merge. Both ran green on every pipeline for months. Neither one ever actually compared anything, because the CI job's base image didn't have git installed, and the script that shelled out to git diff against the merge base failed silently and exited zero anyway.

That's the failure mode worth designing against: not a gate that's wrong, a gate that's vacuous. A wrong gate at least fails loud enough that someone investigates. A vacuous one looks exactly like a working one on every dashboard you'd ever check.

Floors versus manifests

The most common way a gate goes quietly vacuous is a threshold check that only knows how to grow. I had a test-suite count gate set to fail if fewer than 22 suites ran. The actual number of real suites in the repo was 27. Five suites could be deleted entirely and the gate would still pass, because -ge 22 doesn't know what 27 looks like, it only knows what "not fewer than 22" looks like. The fix was replacing the floor with an exact-set manifest: a generated file listing every suite that should exist, diffed against what actually ran. A number in prose about a derived quantity is a claim nothing can check. Put the inventory in a file and diff it.

The same class of bug showed up in a hook test that declared 28 assertions in its own header comment and executed zero of them for the entire life of the file, because the test module was missing an entrypoint guard and nothing ever actually invoked it. It reported success because nothing ran to report failure. I added a separate check that compares declared assertion counts against executed ones, specifically so a test file can't silently stop testing anything while still showing green.

Ratchets instead of allow_failure

allow_failure: true is the standard way to introduce a new check without blocking a repo that already has violations. It also hides every future violation, including brand-new ones, because the whole job is advisory forever unless someone remembers to flip it back. I moved to shrink-only ratchets instead: a check records its current violation count, and the gate fails if that count ever goes up, even by one, while the count is still nonzero. New violations block immediately. The backlog drains on its own schedule without anyone having to watch a dashboard for the moment it's safe to make the gate real.

The image that had no git

The root cause behind the version-bump gate, though, wasn't the gate logic at all. It was the base image. A slim Python image with no git binary meant every git diff call in the job failed, and the wrapping script didn't treat that failure as fatal. apt-get install inside that same runner pod also exited zero while installing nothing, for reasons specific to that runner's networking, which meant even attempts to patch it in place from within the job quietly did nothing either. Two gates that were supposed to be load-bearing had been reporting success for months without ever running their actual check.

The fix was two separate things, deliberately not one: switch to a full image that actually has git, and add a script-level assertion that fails hard when the base ref can't be resolved in a CI context. Fixing the image alone would have left the class of bug intact for the next slim-image migration. The script now refuses to interpret "I couldn't figure out what changed" as "nothing changed."

What this adds up to

None of these are exotic bugs. A missing binary, a floor instead of a set, a test with no entrypoint, an advisory flag left on too long. What they share is the same shape: each one looks identical to a correctly functioning gate from the outside, on every dashboard, in every pipeline summary, right up until you go looking for the specific thing it was supposed to have caught and find that it never could have.

The general fix isn't more gates. It's gates that can fail on their own account, separate from the thing they're checking: a test that plants a known defect and asserts it gets caught, a manifest that has to match exactly instead of merely clearing a bar, a ratchet that can only tighten. A green pipeline tells you a job exited zero. Whether it checked anything is a separate question, and it's worth building something that answers it.