#August 4: Certificate issuance delays from Sidekiq overload (14:45UTC)

A surge of certificate-check and renewal work overwhelmed our background job system, causing significant delays issuing and renewing TLS certificates. This was caused by a few reasons, but the main one being an internal DNS resolver dedicated to the certificate renewal path: it is configured with no caching and aggressively recurses to upstream whenever possible in order to minimize delays in observing DNS updates for customers’ domains. That, however, was initially configured at a time where we were operating at a much smaller scale than we are today, and on this specific day, a spike of DNS renewal tasks combined with possibly slow lookups for some domains completely paralyzed the TLS certificate issuance queue. In the process of investigating this issue, we also discovered that the TLS certificate renewal task has been trying to renew certificates for deleted apps (with no success, of course), which did not help with the excessive queuing delays seen on this day.

We have put in several mitigations for this. Firstly, the DNS resolver used for TLS certificate renewal is now allowed to do some minimal caching to act as a buffer against temporary spikes in TLS renewal tasks and domain resolution failures. This does mean that updated DNS configuration could take a little while to propagate to our certificate issuer, but since this is on the order of seconds, it should not be noticeable under normal operation, and it provided enough protection for us to resolve the Sidekiq queuing issue. We also fixed several issues in our codebase that did not filter out deleted apps correctly, so that we are less likely to experience these spikes in the first place.

This is not the first time we’ve had Sidekiq-related incidents this year. We have plans to improve reliability around certificate issuance – our earlier work to move certificates out of Vault due to numerous incidents being an example – and another part of it is to decouple our certificate issuing pipeline from the rest of our API implementation, a Ruby on Rails app. The hope is that in this process we will make it both more scalable and easier to understand and debug.