#June 4: Stale 6PN mappings wreaking havoc (09:18UTC)

This is another case where, as we were working towards improving the platform, we ended up with multiple ways of doing one thing, some of which are considered legacy and should eventually be removed, but the removal was never completed. An unexpected interaction between the old and new systems then wreaked havoc.

In this case, the system in question is 6PN, our private network powered by Wireguard that connects all of your Machines. When this system was designed, each Machine’s private 6PN address was bound to the host where it was created. This made routing simple to implement, but also started to cause issues when we migrated Machines between different hosts. The reason is that some apps depended on a static 6PN address per Machine: even our own legacy unmanaged Postgres offering depended on it, despite the fact that these addresses were never meant to be stable.

At some point, we finally decided that this is not sustainable and we should, instead, meet the expectation of a majority of apps: that is, to keep 6PN addresses stable. The first iteration of this work was a simple DNAT, where machines still get new 6PN addresses, but an eBPF program rewrites packets targeting a machine’s old 6PN address(es) to the new one. The price we pay is that, because technically the 6PN address still changes, we need to keep track of every single 6PN address a Machine has ever had. This is all stored in Corrosion, which bloated its storage, not to mention the map we needed to synchronize into the eBPF program.

This was changed roughly a year ago. Instead of keeping track of all old 6PN addresses, we simply made it so that Machines do not get reassigned a new 6PN on a new host if it is migrated. All Machines created after this change retain their initial 6PN after migration. Of course, this alone would break routing, because that depended on a per-host fixed 6PN prefix. Some sort of NAT is still needed, but now we only need to keep track of a Machine’s current host and its initial 6PN.

…which brings us to today. We have two types of “stable 6PN” Machines: some before the change above, and some after. The intention was that when an old-style 6PN Machine gets migrated, it will become a new-style stable 6PN Machine with all the new-style plumbing. We’d delete unneeded Corrosion entries in this case and slowly drain them away as they’re moved around. At some point this year, we realized that the Corrosion subscriptions used for old-style 6PN DNAT were creating a lot of load on Corrosion. As a result, we shipped a change to only apply 6PN DNAT rules once when the service responsible for this is started, since we did not except any new Machines to be created with old-style 6PN anymore. However, there was an oversight: in some cases, the existence of old-style DNAT rules actually overrides new-style stable 6PN’s rewriting logic. So, when a Machine gets migrated to use new-style stable 6PN, it is possible that some peers might still be rewriting its address to a host-specific address that no longer exists.

This exact scenario started happening first for our multi-tenant Consul clusters (used for unmanaged Postgres and LiteFS), and then for some customer Machines as a spike of rebalancing migrations happened for various reasons. A considerable amount of time was spent on triaging the issue because it was an unexpected failure mode. We did not expect that old-style stable 6PN would interact with new stable 6PN in this way, especially not several weeks after the last round of changes were deployed.

We mitigated this problem by adding code to delete old-style 6PN DNAT entries when new-style stable 6PN rules are set up. This, unfortunately, briefly caused another bug where the daemon responsible for this became too slow to catch up with Corrosion (we need Corrosion to decide whether a Machine has been migrated and thus needs rules for stable 6PN), which caused issues with Managed Postgres in LAX for a little while. This was then fixed up by making the cleanup code opportunistic and non-blocking for the main processing path.

We see a few directions as the next steps to preventing this from happening again:

1. Old-style 6PN DNAT mappings should really not exist anymore. We need to migrate all the remaining machines that still use it to new-style stable 6PN addresses.
2. The reason why the Corrosion subscription and its processing code became slow was partially due to the query’s inefficiency; we’re working on addressing that too.
3. We need a way to gracefully recover from such an event; the daemon should not just miss updates.
4. Finally, we should be able to “fill in” missing stable 6PN NAT rules even if the Corrosion subscription happened to miss some updates. The subscription can still be used for updates, but not as the single point of failure.