#March 11: GraphQL overloaded (10:26UTC)

Our GraphQL API became overloaded when requests backed up after our secret storage service was temporarily unavailable. Working through the increased backlog of requests led to timeouts and failing health checks for the dashboard and GraphQL endpoints (the Machines API itself remained available). We mitigated the issue about 25 minutes after impact began by scaling up the GraphQL API and increasing its request concurrency.