Broken routing in GRU

#September 24: Broken routing in GRU (16:26UTC)

Routing in our GRU region was degraded for approximately two and a half hours. During this period, changes to app configuration and health probes did not always propagate consistently across the region. As a result, some requests to Machines in GRU returned 503 errors, and some applications experienced inconsistent routing behavior.

Other regions were unaffected, and all existing Machines continued running throughout the incident.

Root Cause

The incident was triggered by a rollout of a new replication metadata format for Corrosion, our distributed state-propagation system, to the GRU regional cluster. A bug in the new code prevented certain updates from replicating correctly between nodes, leaving routing information inconsistent across the cluster. As a result, fly-proxy identified some backends as unhealthy or missing and returned 503 responses.

Resolution

We rolled back the metadata format change, restored the cluster from a backup, and initiated a cluster-wide data synchronization. However, the cluster remained out of sync due to a separate configuration issue that prevented communication between Corrosion nodes.

A recent configuration change had updated the IP address Corrosion uses to listen for peer connections, but the corresponding address registered in Consul, which nodes use to discover one another, had not been updated. This mismatch had gone unnoticed because nodes continued using previously cached peer addresses. Rebuilding the cluster from a backup exposed the issue when nodes attempted to rediscover their peers.

We rolled back the configuration change, immediately restoring communication between nodes. The cluster’s state converged within a few minutes, and routing recovered.

Follow-up Actions

Our existing testing did not catch this issue before it reached production. We are improving our testing and monitoring to help identify similar issues earlier and reduce the likelihood of recurrence. We are also continuing to test the new metadata format before proceeding with a further rollout.