#July 1: Runaway cleanup job filled Redis (05:44UTC)

Our GraphQL API runs its background jobs on Sidekiq. When an app is deleted, we enqueue a cleanup job to tear down any associated resources, and that job in turn enqueues a job for every Machine the app ever had - even ones that had already been deleted.

One app had a very large Machine history. Its cleanup job loaded that entire history into memory at once to enqueue the per-Machine jobs, which pushed the worker’s memory high enough that the supervisor recycled it - killing the process and restarting the job from scratch on another worker. The loop repeated and quickly piled up tens of millions of per-Machine jobs. That filled the disk on the Redis instance backing Sidekiq; once Redis could no longer write its append-only file, job processing stalled and customers saw API errors for around 40 minutes. Already-running Machines were unaffected and kept serving traffic.

We mitigated by extending the Redis volume and restarting Redis, and shipped a change to make the cleanup job safer - it now iterates in smaller batches without pulling everything into memory and skips already-deleted Machines. Once Redis recovered, the backlog drained and error rates returned to normal.