August 13: Stuck nftables reload blocked flyd startup causing API errors

#August 13: Stuck nftables reload blocked flyd startup causing API errors (17:28UTC)

While deploying an nftables change to our worker hosts, a simultaneous flyd (our orchestrator for Fly Machines) deployment happened which saw a number of hosts stuck trying to restart flyd. Because of this, API requests targeted at machines on those hosts were temporarily failing.

flyd was stuck restarting because its systemd service was waiting for nftables to reload, due to Requires=nftables.service and After=nftables.service clauses. These clauses are necessary to ensure that we do not accidentally expose customer VMs to the Internet in case of nftables issues. However, in this case, nftables was taking an abnormally long time to reload, which in turn blocked flyd from restarting in the interim (note: this is due to systemd‘s service startup ordering; in this case, it is not strictly necessary to queue flyd like this since the nftables service was never being fully restarted, only reloaded, but it is how systemd works).

We traced the slow nftables reload down to a suboptimal save/restore script we use for the purpose of machines’ network policies – they are implemented as nftables rules and must be restored on reload. The script to reload these rules was looping through every single machine on a host calling nft list chain repeatedly, which added up to a long time on the more busy hosts. It would not have been a problem if flyd wasn’t being deployed at the same time, either. This incident eventually resolved itself once nftables finished reloading, but we’ll optimize how the reload script works so that this does not happen when we need to reload nftables again. We would like to also optimize the way we use nftables for network policies to reduce the sizes of nftables on worker hosts.