#August 10: Egress IP issues in SJC (06:57UTC)

We saw intermittent outbound connectivity failures for traffic using static egress IP addresses in SJC. The issue turned out to be caused by an unhealthy WireGuard tunnel between a worker host and its egress gateway. This is, in fact, a known bug internally: sometimes our WireGuard tunnels just lose connectivity to one of the peers. Usually this requires a reset of the WireGuard tunnel (fun fact: if you do it wrong, resetting the tunnel might also deadlock the rest of the kernel’s network stack), which we usually will do when we receive an alert for this. Unfortunately, on this day, that specific alert was caught in the middle of another alert storm, and was not noticed and handled for a significant amount of time. We were able to resolve this quickly once we realized this fact, but this incident did tell us a few things we need to work on:

1. The alert and incident handling process. This is something we now have an entire team for, and we are hopeful that ignoring alerts inadvertently during an alert storm will be a thing of the past soon;
2. Reliability of our internal network mesh (WireGuard) itself. This bug is already fixed upstream, so any new hosts we have provisioned recently will not see the same issue. For existing hosts, kernel upgrades will be applied as they are gradually rebooted. Of course, this is a somewhat slow process and unless we decided to reboot every existing host at once, we’ll likely continue to see these issues from time to time. Fortunately, we have been working on the ability to reroute traffic over the WireGuard mesh when this kind of connectivity issues happen, including but not limited to this WireGuard bug. Once that work is completed, hopefully these occasional issues will cause much less disruption from users’ perspective.