The credential nobody watched

An internal service I run brokers write access to Jira on behalf of a fleet of AI tools, using one shared service-account API token. That token expired, silently, and took the integration down for more than 40 hours. Atlassian caps API tokens at 364 days, doesn't auto-renew them, and doesn't expose an API to check when one is about to lapse. Nobody had rotated it since it was created. By the time anyone noticed, it had been dead for a while.

The instinct after an outage like that is to fix the rotation cadence. That's not what actually mattered here. What mattered was that the failure surfaced as one HTTP error code that could mean four completely different things, and nobody could tell which one they were looking at without guessing.

One error, four causes

The failure showed up as a 400 on issue creation with a permission-denied message. That single error code and message can mean the request named the wrong project key, the service account is missing a permission grant on the right project, the account has no product seat at all, or the payload referenced an issue type by name instead of by id. All four produce the identical response. From outside the vendor's system there is no way to distinguish a token that's dead from a permission that's missing from a typo in a field.

That ambiguity is a design defect in the client, not an annoyance to shrug off as "the vendor's API is like that." If your service can afford two extra idempotent read calls on a failure path, spending them to disambiguate the cause is worth more than any amount of retry tuning, because retrying a request that's failing for a reason retries can't fix just burns time while the actual diagnosis sits undone. The fix was a self-diagnosing probe that fires on a project-error response: two read-only calls that check the project exists, the account can see it, and the account holds the right permission, and logs exactly which one came back wrong. What used to be "something's broken, go find out what" became a log line that names the cause.

Silence is not proof of health

The second piece was a monitor that pages the team 45 days ahead of the token's known expiry date, which sounds obvious until you notice the failure mode hiding inside it: if the environment variable holding that expiry date is simply unset, the monitor doesn't fire, doesn't warn, doesn't do anything, and looks from the outside exactly like a healthy system that has nothing to report. An alert that depends on a config value being present has a second, quieter way to fail: the config value being absent. That's worth flagging as its own watched condition, not trusted as an edge case that surely won't happen.

A second, unrelated bug that showed up during the same incident

While this was being diagnosed, Jira's error messages switched from English to another language mid-incident, with no deploy on either side. The account's locale setting had rotated on its own, and Jira ties error-message language to account profile locale rather than to the request. Log search that had been built around matching English error text stopped matching anything, during the exact window it mattered most. The fix looks like a one-header change and isn't: sending Accept-Language alone gets silently ignored, and you also need Jira's own X-Force-Accept-Language header set to force it. The tempting one-line fix looks correct in testing and quietly doesn't work in production, which is its own small lesson about verifying a header fix against the vendor's actual behavior rather than against what the header's name implies it should do.

What I'd actually recommend

Rotate your credentials, obviously. But the return on that investment is small compared to making your own client legible when something upstream does fail: disambiguate an ambiguous error instead of retrying blind, and build your health signal so an unset config value reads as a failure to alert, not as nothing to report. A shared credential going stale is not primarily a schedule problem. It's an observability problem wearing a rotation problem's clothes, and the fix that actually shortened the next incident was the one aimed at the second thing, not the first.