#March 6: App-scoped IPs deleted by mistake (04:11UTC)

Our egress IPs are assigned in regional blocks. A while ago, when we deprecated some regions, these blocks and assignments were migrated to the new destinations. On this day, an automated egress IP reseed reintroduced stale “unassigned” egress IP records for a few deprecated regions; these were then synchronized into regional Corrosion clusters, which makes use of our fly-force-region header through the Fly Proxy, which itself rewrites deprecated regions to their migration destinations. This inadvertently deleted the assigned app-scoped egress IP entries which were migrated to the new regions. Affected apps temporarily lost the expected app-scoped egress IP, until we manually re-synced the assigned egress IPs. We then removed the stale deprecated-region records and added safeguards to keep deprecated regions out of the reseed path and to harden internal regional-routing requests so this can’t recur.