We have an ancient GraphQL API, which both flyctl and our dashboard rely on in certain ways.

Last week, we were seeing elevated 503s when deploying the app that serves this API. The reason turned out to be that we were using the default max_unavailable with the rolling deploy strategy.

That default is 0.33, i.e. one third of the fleet could be unavailable at any point during the deploy. This caused the rest of the fleet to receive a lot of extra traffic. Some Machines were able to cope and some weren’t. Those that weren’t ended up failing their health checks, meaning the HAProxy instance that sits in front of the API app marked it as down and started returning 503s. We have multiple HAProxy Machines, and not all of them saw health checks fail, so some continued proxying requests to the API.

The solution here was simple enough - we changed max_unavailable to 0.05, and since then we haven’t seen this failure mode cause any API unavailability.

The other problem we found was with an app that services a Sidekiq job queue. This app was also using the default rolling strategy with max_unavailable = 0.33.

To compound the issue, we also had no health checks on this app. That makes some conceptual sense, since it’s not serving requests. Without health checks, flyctl considers a Machine as “deployed” once it has been running for 10 seconds without crash-looping.

The combination of these two meant that this app was deploying all of its Machines within about 60 seconds. Each Machine made a fair number of database connections at boot, and each connection caused a temporary load on our database. When you add it all up, we were overloading our primary database, meaning anything that interacted with the database (various APIs/our dashboard/our job queue workers) could be slowed/time out.

The fix here was two-pronged:

We also reduced max_unavailable, but only to 0.15, since we don’t need to maintain quite as much availability for job queue workers. This was more about pacing the deploy to ease database load due to new connections rather than maintaining availability.
We added a health check, so that the deploy couldn’t rapidly blitz through all Machines. Without this health check, reducing max_unavailable wouldn’t have done very much for us.

Since these fixes, we haven’t seen any HAProxy instances mark the API as unavailable, and the number of proxy errors is significantly reduced. We’ll always have some proxy errors because rolling doesn’t cordon Machines before updating them. It’s definitely got me interested in adding something like a rolling-cordon strategy!

The tradeoff to these changes is that our deploys are a bit slower, but certainly not unbearably so.