---
title: A green pipeline is not evidence
description: Two load-bearing CI gates ran green for months without comparing anything, because the base image had no git and nobody noticed the difference between passing and not running.
date: 2026-09-22
area: development
tags:
  - patterns
  - devex
  - build
draft: false
---

Two gates in one of my repos existed specifically to catch a version-bump and changelog mismatch
before merge. Both ran green on every pipeline for months. Neither one ever actually compared
anything, because the CI job's base image didn't have git installed, and the script that shelled
out to `git diff` against the merge base failed silently and exited zero anyway.

That's the failure mode worth designing against: not a gate that's wrong, a gate that's vacuous. A
wrong gate at least fails loud enough that someone investigates. A vacuous one looks exactly like a
working one on every dashboard you'd ever check.

## Floors versus manifests

The most common way a gate goes quietly vacuous is a threshold check that only knows how to grow.
I had a test-suite count gate set to fail if fewer than 22 suites ran. The actual number of real
suites in the repo was 27. Five suites could be deleted entirely and the gate would still pass,
because `-ge 22` doesn't know what 27 looks like, it only knows what "not fewer than 22" looks
like. The fix was replacing the floor with an exact-set manifest: a generated file listing every
suite that should exist, diffed against what actually ran. A number in prose about a derived
quantity is a claim nothing can check. Put the inventory in a file and diff it.

The same class of bug showed up in a hook test that declared 28 assertions in its own header
comment and executed zero of them for the entire life of the file, because the test module was
missing an entrypoint guard and nothing ever actually invoked it. It reported success because
nothing ran to report failure. I added a separate check that compares declared assertion counts
against executed ones, specifically so a test file can't silently stop testing anything while
still showing green.

## Ratchets instead of allow_failure

`allow_failure: true` is the standard way to introduce a new check without blocking a repo that
already has violations. It also hides every future violation, including brand-new ones, because
the whole job is advisory forever unless someone remembers to flip it back. I moved to
shrink-only ratchets instead: a check records its current violation count, and the gate fails if
that count ever goes up, even by one, while the count is still nonzero. New violations block
immediately. The backlog drains on its own schedule without anyone having to watch a dashboard for
the moment it's safe to make the gate real.

## The image that had no git

The root cause behind the version-bump gate, though, wasn't the gate logic at all. It was the base
image. A slim Python image with no git binary meant every `git diff` call in the job failed, and
the wrapping script didn't treat that failure as fatal. `apt-get install` inside that same runner
pod also exited zero while installing nothing, for reasons specific to that runner's networking,
which meant even attempts to patch it in place from within the job quietly did nothing either.
Two gates that were supposed to be load-bearing had been reporting success for months without ever
running their actual check.

The fix was two separate things, deliberately not one: switch to a full image that actually has
git, and add a script-level assertion that fails hard when the base ref can't be resolved in a CI
context. Fixing the image alone would have left the class of bug intact for the next slim-image
migration. The script now refuses to interpret "I couldn't figure out what changed" as "nothing
changed."

## What this adds up to

None of these are exotic bugs. A missing binary, a floor instead of a set, a test with no
entrypoint, an advisory flag left on too long. What they share is the same shape: each one looks
identical to a correctly functioning gate from the outside, on every dashboard, in every pipeline
summary, right up until you go looking for the specific thing it was supposed to have caught and
find that it never could have.

The general fix isn't more gates. It's gates that can fail on their own account, separate from the
thing they're checking: a test that plants a known defect and asserts it gets caught, a manifest
that has to match exactly instead of merely clearing a bar, a ratchet that can only tighten. A
green pipeline tells you a job exited zero. Whether it checked anything is a separate question, and
it's worth building something that answers it.