The prompt-cache floor that made the cheap model expensive

I built an AI-assisted QA plugin at KnowBe4 that reviews e-learning course packages before they ship, and one stage of it grades keyframes pulled from a rendered course against a shared rubric. The obvious cost architecture for that stage is a two-tier one: a cheap, fast model triages every frame, and only the frames that trip a finding, an explicit escalation flag, or a confidence score below threshold get sent to a more expensive model for a second look. Most frames are clean, so most of the volume should run on the cheap tier. The design only pencils out, though, if the cheap tier's cost per call is actually cheap, and that depends on something the architecture diagram doesn't show at all.

The assumption baked into the diagram

The triage call sends the same rubric text on every single frame, roughly 1,500 tokens of instructions that don't change from one call to the next. Prompt caching exists for exactly this case: pay full price the first time, then pay a fraction of that price on every subsequent call that reuses the same prefix. The cost model for the whole two-tier design assumed that rubric was being cached. It had to be, or the "cheap" tier wasn't actually cheap, it was just a smaller sticker price on the same full-price call repeated thousands of times a week.

Nothing in the pipeline verified that assumption. The request set cache_control on the shared prefix, the shape of the call looked correct, and the architecture was designed around caching working. Designed around is not the same as measured.

What the token counts actually said

Reading the raw usage data per call told a different story. Every triage call carried 2,243 input tokens, of which the shared rubric accounted for 1,495, about 67 percent of the request. The cache_read_input_tokens field, the number that says how many of those tokens came from cache instead of full price, read zero. Not low. Zero, on every single run ever recorded for that stage. The cheap tier had never once served the shared prompt from cache, not on the first call and not on the ten-thousandth.

The reason turned up in the model's own documented limits. That model's minimum cacheable prefix is 4,096 tokens. The shared rubric was 1,495. The request was well under the floor the model requires before it will cache anything at all, so the API silently accepted the caching directive on every call and did nothing with it. No error, no warning, no field that flags a caching request as rejected. It just billed full price and returned a normal-looking response, indistinguishable from a cache hit unless you go looking at the token accounting yourself.

Why the fix wasn't a bigger prompt

The reflexive fix is to pad the rubric past the floor: add filler text, restructure the prompt, whatever gets the shared prefix over 4,096 tokens so the same cheap model starts caching. I considered it and rejected it. Inflating a rubric to satisfy a caching threshold couples prompt content to a billing implementation detail, and the next rubric edit could shrink it back under the floor without anyone noticing until the telemetry was checked again.

The more capable model in the pipeline has a minimum cacheable prefix of 1,024 tokens, comfortably under the 1,495-token rubric as written. Running triage on that model instead meant the shared prompt cached correctly from the first call. On one module, switching triage tiers took cache reads from 0 to 23,920 tokens per run and cut total run input tokens by about 25 percent, even while output tokens rose because the more capable model reasons more and finds more: four visual findings where the cheap tier had found one. At the plugin's planned weekly volume, the difference works out to roughly $130 to $156 a week at list pricing, and that number comes from real per-frame token measurements scaled to a volume target, not a hypothetical.

Verification as an ongoing practice

The interesting failure here isn't a bug in the request. cache_control was set correctly, the architecture was sound on paper, and every unit test that checks the shape of a call would have passed. The gap only exists in a field most of the pipeline's own logging never printed. A two-tier cost architecture makes exactly one load-bearing assumption that the diagram can't express: that the cheap tier's repeated context clears whatever cache floor that specific model enforces. That floor differs by model and isn't obvious from pricing pages built around per-token rates, not caching behavior.

So checking cache_read_input_tokens against expected volume is now a standing check on any stage in that pipeline that claims to use caching, not a one-time audit. Every stage that sets cache_control reports its cache-read count in its own output alongside the findings it produces, specifically so a silent zero shows up in the normal run log instead of requiring a special investigation to surface. A cost model is a claim about what the API will actually do with a request. The only way to know if that claim holds is to read what the API says it did, not what the architecture intended.