#July 16: Vault throws one last tantrum on its way out the door (11:28UTC)

Recently we have been migrating our certificate storage away from Vault to our Petsem codebase, to reduce the number of distinct cluster-shaped things we need to think about. We reached the 100% mark on the rollout a few days before this incident, so this body of work was all but done.

This is, of course, when Vault decided to break in some manner. Since we were still validating things and hadn’t rolled out our code sans feature flags yet, there remained calls to Vault in both our GraphQL API and the Fly Proxy, particularly as a fallback for unknown certificates. Though these weren’t in the hot path of healthy requests, Vault hanging caused some issues across the services, and caused TLS handshakes to fail for otherwise functional Fly Apps for some number of minutes.

Unlike most writeups here, we don’t have (or, need) a firm root cause on exactly what broke and where. As soon as this kicked off we made the call to rip out the Vault functionality, since that was up next anyway. Without Vault in play, we got to blissfully ignore whatever went wrong with it. And we didn’t have much of a reason to dig into exactly which Vault/Petsem interaction in the proxy misbehaved, as the code that housed it was razed.