#March 2: Our certs vault went down and took some proxies with it (20:09UTC)

A partial update to Vault, which we use to store TLS certificates, left it in a state where requests would hang for several minutes. In the face of most Vault issues, the Fly Proxy will continue with the cached certs it has on hand, limiting impact only to those hostnames that are new or have not been served recently. Annoyingly, Vault wasn’t down down, it was just slow. As our proxies encountered hostnames with uncached certificates, a growing pile of requests stalled on the hung-but-not-failing Vault lookup. Eventually there were enough requests in this state on some proxies that we simply ran out of connection slots, causing widespread connection errors. Service recovered after a few minutes once Vault finished its update.