September 2: Sprites API partial outage from SQLite database evictions

#September 2: Sprites API partial outage from SQLite database evictions (22:54UTC)

Sprites API stores per-org state in lightweight SQLite databases, each managed by a local (to the organization) control plane node. The nodes periodically write per-database heartbeats to object storage to prove they’re alive. If a node misses a heartbeat, it assumes something is wrong and evicts all of its databases, which then attempt to restore from their Litestream replicas on other nearby nodes.

For about 30 minutes, multiple control plane nodes started failing their heartbeat writes. The failures hit nodes across several regions but were primarily concentrated in Europe. Each failed heartbeat triggered a bulk eviction, displacing hundreds of databases to other failing nodes. Databases continued to hop between nodes until heartbeats resumed.

We’re still not sure exactly what caused the heartbeat writes to start failing. Our leading hypothesis is intermittent connectivity loss between our nodes and Tigris (our object storage provider), amplified by a bug in the retry logic that made subsequent attempts always fail.