#August 20: Development node advertised Fly public DNS anycast IPs and broke it (20:08UTC)

An internal development server was misconfigured to advertise our public DNS’s IP address, but the node’s DNS service had been failing. As a result, some recursive resolvers routed queries to that location and experienced brief DNS resolution errors. We resolved this by removing that server from the anycast/DNS serving range so it stopped receiving DNS traffic.

Of course, the most obvious answer here going forward is that this kind of misconfiguration should not happen again, and we need to have more aggressive alerts about misconfigured DNS. Another route of improvement is to make our two DNS anycast IPs actually redundant to each other by announcing them from different subsets of PoPs.