September 24: Network readiness checks stalled worker hosts during reroute

#September 24: Network readiness checks stalled worker hosts during reroute (09:46UTC)

As we covered in a recent infra log, we have been working hard on improvements to our 6PN private networking this year. The goals for this work were to improve reliability and unlock some new tricks, one of which is 6PN rerouting.

The gist is that, with the added flexibility in our internal routing, we can now reroute 6PN traffic to avoid network degradation. When our monitoring detects loss between two datacenters, we route 6PN traffic through an intermediary datacenter with a healthy connection to both ends, if one exists. This is something we have been doing for a couple of months already, we just haven’t talked about it much. The good news is that it has made our platform more resilient to upstream networking issues, and private networking between regions has been a lot more reliable since.

In this incident, our network monitoring detected packet loss into our Mumbai (BOM) region from a range of other datacenters. In response, our reroute system kicked in and selected datacenters in Singapore (SIN) as relays for much of the traffic. A few hundred hosts worldwide switched their Mumbai-bound traffic paths to SIN at roughly the same time. The reroutes themselves were fine, but every source host first probed the candidate relay workers with a readiness check to confirm they could forward traffic. That check did a full dump of all network interfaces and addresses via netlink on the target host, which serialized behind a global kernel lock. Each dump took about 500ms on a busy host, so when 100+ source hosts hit the same relay worker simultaneously, we instantly queued up over 50 seconds of serialized kernel work. This, unsurprisingly, caused broad issues, and the impacted hosts went dark while connections stalled. A similar pattern hit workers in LAX about six hours later.

The fix was straightforward: we single-flighted the readiness check so concurrent requests share a result, and replaced the full netlink interface/address dumps with a targeted lookup filtered to the one interface we actually care about. Response time for the check dropped from 511ms to 29ms, and 100 concurrent requests no longer duplicate the check.