October 12: Expired TLS certificate on sprites.app

#October 12: Expired TLS certificate on sprites.app (00:38UTC)

Hosted Sprites on *.sprites.app were unavailable for roughly 75 minutes, due to a bug in our certificate provisioning meeting a bug in the Fly Proxy.

We issue both RSA and ECDSA certs for domains via Let’s Encrypt, and renew both of these in lockstep. When the certs came up for renewal about a month before the outage, the RSA cert renewed successfully but the ECDSA renewal either silently failed or never ran. We’d since found and fixed this bug, but the Sprites cert was renewed before the fix shipped, leaving it with a fresh RSA cert and a soon-to-expire ECDSA cert.

When the ECDSA cert expired, our proxy should have fallen back to the still-valid RSA cert. That fallback worked well for errors and absent certs, but when it “successfully” found an expired cert it bailed in a rather unhelpful way.

Bonus problem: We lacked automated alerting for Sprite URLs, so we didn’t notice the outage as quickly as we should have. Once the incident was declared, we reissued the missing cert within 10 minutes.

The proxy has since been fixed to prevent this from happening again, and we’re continually improving our alerting on the new Sprites surfaces.