June 11: Everyone lives on NULL island
#June 11: Everyone lives on NULL island (06:03UTC)
Newly created WireGuard peers and Depot builders were placed in AMS (Amsterdam) regardless of where the user actually was. We had removed some datacenter coordinate metadata from Consul, believing it to be unused – but part of our control plane still sourced region coordinates from it, and without that data it located every region at (0,0). The logic picks the region nearest the user, but with distances all tied at zero it fell through to the alphabetically first region, AMS. Existing peers and builders were unaffected; only newly placed ones landed in the wrong region.
We mitigated this by sourcing region coordinates from our database instead of Consul, then cleaned up the Depot builders that had been pinned to AMS. We’ve also hardened the placement logic to fail loudly if coordinates go missing in the future.