September 23: Sprites API outage during database migration

#September 23: Sprites API outage during database migration (14:10UTC)

Sprites API operations, including creating, listing, and restarting sprites, were unreliable for about 7 hours and 43 minutes. This happened off the back of a migration to PostgreSQL, which activated background database processes for thousands of previously inactive organizations. A metrics collector running every 15 seconds kept related cluster connections active, while a replication bug caused unnecessary data transfers between nodes. Together, these produced enough disk and network load to disrupt the service.

We tried several mitigations to reduce background work and improve recovery, but they did not stop the underlying problem. We also tested a different hypervisor while investigating a possible I/O issue, then reverted when performance got worse. The main fixes were to start database processes only when needed, disable the metrics collector, and correct the replication bug. We have since replaced the collector with one that reads local state without opening cluster connections, added monitoring for disk and network pressure, and removed the old database code after a move to PostgreSQL.