#July 14: SJC worker hosts locked up and rebooted (14:53UTC)

While debugging network packet loss in SJC, a tc qdisc (queuing discipline) configuration update was (unnecessarily) pushed out to all of SJC workers, which caused a number of them to lock up. We promptly started rebooting them and nursing them back to health.

Honestly, we are still not sure why this happened in the first place. The tc qdisc configuration was, after further validation, later very carefully deployed to all of our hosts in all regions, and none experienced a similar problem whatsoever. qdisc is not supposed to just randomly lock up entire systems: at most, it should cause network issues that we can easily recover from. So, it seems likely that we must have hit some weird kernel or NIC driver bug, but so far we do not know exactly which one. One thing is for sure though: we’ll have to be much more careful about even the most unassuming changes related to NICs.