---
title: "Stuck nftables reload blocked flyd startup causing API errors"
---

# Stuck nftables reload blocked flyd startup causing API errors

[← View full Infra Log](https://fly.io/infra-log/)

---

August 13: Stuck nftables reload blocked flyd startup causing API errors (17:28UTC)

While deploying an `nftables` change to our worker hosts, a simultaneous `flyd` (our orchestrator for Fly Machines) deployment happened which saw a number of hosts stuck trying to restart `flyd`. Because of this, API requests targeted at machines on those hosts were temporarily failing.

`flyd` was stuck restarting because its `systemd` service was waiting for `nftables` to reload, due to `Requires=nftables.service` and `After=nftables.service` clauses. These clauses are necessary to ensure that we do not accidentally expose customer VMs to the Internet in case of `nftables` issues. However, in this case, `nftables` was taking an abnormally long time to reload, which in turn blocked `flyd` from restarting in the interim (note: this is due to `systemd`‘s service startup ordering; in this case, it is not *strictly necessary* to queue `flyd` like this since the `nftables` service was never being fully restarted, only reloaded, but it is how `systemd` works).

We traced the slow `nftables` reload down to a suboptimal save/restore script we use for the purpose of machines’ network policies – they are implemented as `nftables` rules and must be restored on reload. The script to reload these rules was looping through every single machine on a host calling `nft list chain` repeatedly, which added up to a long time on the more busy hosts. It would not have been a problem if `flyd` wasn’t being deployed at the same time, either. This incident eventually resolved itself once `nftables` finished reloading, but we’ll optimize how the reload script works so that this does not happen when we need to reload `nftables` again. We would like to also optimize the way we use `nftables` for network policies to reduce the sizes of `nftables` on worker hosts.