September 2: Sidekiq periodic scheduler not running after Redis timeout

#September 2: Sidekiq periodic scheduler not running after Redis timeout (07:15UTC)

Sidekiq uses a leader election to decide which worker process enqueues periodic (aka cron) jobs. The newly elected leader hit a transient network issue communicating with Redis at exactly the wrong moment: the process still became the elected scheduler, but it never completed initialization of its periodic job manager. That left scheduled background work not being enqueued for roughly five hours. The missed jobs included billing usage aggregation, certificate renewal checks, and rechecks for newly requested certificates that had not passed their initial domain validation.

Once we detected the problem, we restarted the wedged scheduler process to restore periodic job execution, and backfilled the missed billing usage.

We’re adding direct monitoring for missed periodic jobs so it doesn’t take us five hours to notice, and improving the reliability of the network between the Sidekiq workers and Redis.