---
title: The prompt-cache floor that made the cheap model expensive
description: A two-tier triage pipeline assumed its cheap model's shared prompt was cached. Real token telemetry showed it never once was.
date: 2026-09-22
area: ai
tags:
  - architecture
  - ai
  - patterns
draft: false
---

I built an AI-assisted QA plugin on the job that reviews published content packages before they
ship, and one stage of it grades keyframes pulled from a rendered package against a shared rubric.
The obvious cost architecture for that stage is a two-tier one: a cheap, fast model triages every
frame, and only the frames that trip a finding, an explicit escalation flag, or a confidence score
below threshold get sent to a more expensive model for a second look. Most frames are clean, so
most of the volume should run on the cheap tier. The design only pencils out, though, if the cheap
tier's cost per call is actually cheap, and that depends on something the architecture diagram
doesn't show at all.

## The assumption baked into the diagram

The triage call sends the same rubric text on every single frame, roughly 1,500 tokens of
instructions that don't change from one call to the next. Prompt caching exists for exactly this
case: pay full price the first time, then pay a fraction of that price on every subsequent call
that reuses the same prefix. The cost model for the whole two-tier design assumed that rubric was
being cached. It had to be, or the "cheap" tier wasn't actually cheap, it was just a smaller sticker
price on the same full-price call repeated thousands of times a week.

Nothing in the pipeline verified that assumption. The request set `cache_control` on the shared
prefix, the shape of the call looked correct, and the architecture was designed around caching
working. Designed around is not the same as measured.

## What the token counts actually said

Reading the raw usage data per call told a different story. Every triage call carried 2,243 input
tokens, of which the shared rubric accounted for 1,495, about 67 percent of the request. The
`cache_read_input_tokens` field, the number that says how many of those tokens came from cache
instead of full price, read zero. Not low. Zero, on every single run ever recorded for that stage.
The cheap tier had never once served the shared prompt from cache, not on the first call and not on
the ten-thousandth.

The reason turned up in the model's own documented limits. That model's minimum cacheable prefix is
4,096 tokens. The shared rubric was 1,495. The request was well under the floor the model requires
before it will cache anything at all, so the API silently accepted the caching directive on every
call and did nothing with it. No error, no warning, no field that flags a caching request as
rejected. It just billed full price and returned a normal-looking response, indistinguishable from
a cache hit unless you go looking at the token accounting yourself.

## Why the fix wasn't a bigger prompt

The reflexive fix is to pad the rubric past the floor: add filler text, restructure the prompt,
whatever gets the shared prefix over 4,096 tokens so the same cheap model starts caching. I
considered it and rejected it. Inflating a rubric to satisfy a caching threshold couples prompt
content to a billing implementation detail, and the next rubric edit could shrink it back under the
floor without anyone noticing until the telemetry was checked again.

The more capable model in the pipeline has a minimum cacheable prefix of 1,024 tokens, comfortably
under the 1,495-token rubric as written. Running triage on that model instead meant the shared
prompt cached correctly from the first call. On one module, switching triage tiers took cache reads
from 0 to 23,920 tokens per run and cut total run input tokens by about 25 percent, even while
output tokens rose because the more capable model reasons more and finds more: four visual findings
where the cheap tier had found one. At the plugin's planned weekly volume, the difference works out to
roughly $130 to $156 a week at list pricing, and that number comes from real per-frame token
measurements scaled to a volume target, not a hypothetical.

## Verification as an ongoing practice

The interesting failure here isn't a bug in the request. `cache_control` was set correctly, the
architecture was sound on paper, and every unit test that checks the shape of a call would have
passed. The gap only exists in a field most of the pipeline's own logging never printed. A two-tier
cost architecture makes exactly one load-bearing assumption that the diagram can't express: that
the cheap tier's repeated context clears whatever cache floor that specific model enforces. That
floor differs by model and isn't obvious from pricing pages built around per-token rates, not
caching behavior.

So checking `cache_read_input_tokens` against expected volume is now a standing check on any stage
in that pipeline that claims to use caching, not a one-time audit. Every stage that sets
`cache_control` reports its cache-read count in its own output alongside the findings it produces,
specifically so a silent zero shows up in the normal run log instead of requiring a special
investigation to surface. A cost model is a claim about what the API will actually do with a
request. The only way to know if that claim holds is to read what the API says it did, not what the
architecture intended.