#July 9: A couple of deployment issues (15:11UTC)

This day saw 2 separate instances of deployment-related issues centered around IAD happening around the same time.

The first one was related to Depot builders, where flyctl deploy would wait seemingly indefinitely for a Depot builder to become available. Initially, we believed that this was related to a single IAD host under immense I/O pressure, and was only affecting a subset of Depot builders with backing volumes on that host. Failed hosts with Depot volumes can sometimes prevent existing Depot builders from being reused until recreated via an explicit flyctl --recreate-builder flag. However, after we resolved the host issue (which took a considerable amount of time), it became clear that the Depot issue was not limited to the one host, but was impacting builds for any IAD builders. This is when an internal incident was declared and a status page posted.

It turns out that this was another case of capacity-related Depot issue. Because machines used for Depot builders are pretty large, when we’re under capacity constraints, Depot builders may not be able to successfully start with an existing volume, because that host has no spare capacity to run the builder machine. Usually, machines can auto-migrate on start when capacity error happens, but this is explicitly not enabled for machines and volumes as large as builders. Under these circumstances, no new Depot builder should land in IAD, but existing ones will continue to be attempted and fail to start. The fix here, before adding more capacity, was to purge existing Depot builders from IAD in order to have them recreated in other North American regions as needed.

Separately, our Docker registry, which runs as a Fly App managed by us, started throwing a lot of 5xx errors, while the Depot issue above was happening. This accounted for another bulk of deployment issues happening on this day, which unfortunately compounded with Depot-related issues above. Initially we thought the host DNS resolvers were acting up, since the registry app is throwing a lot of DNS-related errors. After ruling that out, we realized that even pinging the host (fdaa::3, where the DNS resolver also lives) from these machines was showing latency up to 100ms – definitely too much for basically loopback traffic! At first, we thought this is due to CPU starvation since the machines were all at 100% utilization. Scaling up just the CPU did bring down the latency, but it was still at several 10s of milliseconds. Some further investigation made us realize that these machines are running a lot of traffic over their TAP interfaces, and we do know that our TAP interfaces, for one reason or another, starts to struggle a bit under higher ultilization (> 1Gbps). Creating more machines for the registry to spread the traffic out immediately solved the issue.

We came out of this incident with some plans for improvement:

1. We should know about widespread deployment issues way sooner than we did on this day. We did not put up a statuspage because we assumed that the issue was related to a downed host, which already had its separate host issue, but it was in fact unrelated. We now have better monitoring for Depot issues related to machine and volume placement, and we will hopefully be paged much, much sooner should something like this happen again.
2. Even when there is capacity pressure, ideally, Depot builders should still be available, just placed in different regions, for new or existing users. We are working with Depot to add support for placing builders using geo region aliases, which allows much more flexible machine placement, and we are planning to integrate this with our API and flyctl so that local capacity issues no longer automatically translate to much wider deployment failures.
3. For registry, besides similar alerting, figuring out the performance bottleneck on our TAP interfaces is on our plan as well. Fixing this will improve experience across all machines, with the caveat that we’ll likely still place constraints to ensure fairness between machines on a host.