#July 20: tkdb primary host outage caused token validation failures (07:07UTC)

The host running the primary node of tkdb, our token validation service, went offline due to maintenance and did not manage to reboot successfully due to a separate hardware issue. This caused token validation to time out, which in turn led to widespread failures in API operations that depend on authentication (Machines API, flyctl, etc.) and elevated 500s from platform services. API and dashboard functionality recovered over a few minutes once we restored the host to full service.

The root cause of this incident is, of course, the host failure, but a huge contributing factor is that tkdb is single-primary, and, unlike petsem (which powers app secrets and TLS certs), read replicas of tkdb do not function if the primary node is entirely offline. This is because, as a token validation service, the read replicas must be able to receive notifications about revocation to uphold security guarantees. We are currently looking into ways we can relax this requirement, keeping replicas up during primary outages, while avoiding impact on security properties provided by tokens.