---
title: The credential nobody watched
description: A shared service-account token expired silently and took a production integration down for 40 hours. The fix that mattered wasn't rotation. It was making four causes of the same error distinguishable.
date: 2026-09-22
area: infrastructure
tags:
  - aws
  - patterns
  - devex
draft: false
---

An internal service I run brokers write access to Jira on behalf of a fleet of AI tools, using one
shared service-account API token. That token expired, silently, and took the integration down for
more than 40 hours. Atlassian caps API tokens at 364 days, doesn't auto-renew them, and doesn't
expose an API to check when one is about to lapse. Nobody had rotated it since it was created. By
the time anyone noticed, it had been dead for a while.

The instinct after an outage like that is to fix the rotation cadence. That's not what actually
mattered here. What mattered was that the failure surfaced as one HTTP error code that could mean
four completely different things, and nobody could tell which one they were looking at without
guessing.

## One error, four causes

The failure showed up as a 400 on issue creation with a permission-denied message. That single
error code and message can mean the request named the wrong project key, the service account is
missing a permission grant on the right project, the account has no product seat at all, or the
payload referenced an issue type by name instead of by id. All four produce the identical response.
From outside the vendor's system there is no way to distinguish a token that's dead from a
permission that's missing from a typo in a field.

That ambiguity is a design defect in the client, not an annoyance to shrug off as "the vendor's
API is like that." If your service can afford two extra idempotent read calls on a failure path,
spending them to disambiguate the cause is worth more than any amount of retry tuning, because
retrying a request that's failing for a reason retries can't fix just burns time while the actual
diagnosis sits undone. The fix was a self-diagnosing probe that fires on a project-error response:
two read-only calls that check the project exists, the account can see it, and the account holds
the right permission, and logs exactly which one came back wrong. What used to be "something's
broken, go find out what" became a log line that names the cause.

## Silence is not proof of health

The second piece was a monitor that pages the team 45 days ahead of the token's known expiry date,
which sounds obvious until you notice the failure mode hiding inside it: if the environment
variable holding that expiry date is simply unset, the monitor doesn't fire, doesn't warn, doesn't
do anything, and looks from the outside exactly like a healthy system that has nothing to report.
An alert that depends on a config value being present has a second, quieter way to fail: the
config value being absent. That's worth flagging as its own watched condition, not trusted as an
edge case that surely won't happen.

## A second, unrelated bug that showed up during the same incident

While this was being diagnosed, Jira's error messages switched from English to another language
mid-incident, with no deploy on either side. The account's locale setting had rotated on its own,
and Jira ties error-message language to account profile locale rather than to the request. Log
search that had been built around matching English error text stopped matching anything, during
the exact window it mattered most. The fix looks like a one-header change and isn't: sending
`Accept-Language` alone gets silently ignored, and you also need Jira's own
`X-Force-Accept-Language` header set to force it. The tempting one-line fix looks correct in
testing and quietly doesn't work in production, which is its own small lesson about verifying a
header fix against the vendor's actual behavior rather than against what the header's name implies
it should do.

## What I'd actually recommend

Rotate your credentials, obviously. But the return on that investment is small compared to making
your own client legible when something upstream does fail: disambiguate an ambiguous error instead
of retrying blind, and build your health signal so an unset config value reads as a failure to
alert, not as nothing to report. A shared credential going stale is not primarily a schedule
problem. It's an observability problem wearing a rotation problem's clothes, and the fix that
actually shortened the next incident was the one aimed at the second thing, not the first.