---
title: "Reducing deploy and dashboard failures"
description: "Slower, paced rolling deploys and a new health check stopped the 503s and database overloads behind recent deploy and dashboard failures."
date: "2026-11-05"
author: "Jacob Fenton"
url: "https://fly.io/changelog/reducing-deploy-and-dashboard-failures/"
discussion: "https://community.fly.io/t/reducing-deploy-and-dashboard-failures/29003"
---

# Reducing deploy and dashboard failures

We have an ancient GraphQL API, which both `flyctl` and our dashboard rely on in certain ways.

Last week, we were seeing elevated 503s when deploying the app that serves this API. The reason turned out to be that we were using the default `max_unavailable` with the `rolling` deploy strategy.

That default is `0.33`, i.e. one third of the fleet could be unavailable at any point during the deploy. This caused the rest of the fleet to receive a lot of extra traffic. Some Machines were able to cope and some weren't. Those that weren't ended up failing their health checks, meaning the HAProxy instance that sits in front of the API app marked it as down and started returning 503s. We have multiple HAProxy Machines, and not all of them saw health checks fail, so some continued proxying requests to the API.

The solution here was simple enough - we changed `max_unavailable` to `0.05`, and since then we haven't seen this failure mode cause any API unavailability.

The other problem we found was with an app that services a Sidekiq job queue. This app was also using the default `rolling` strategy with `max_unavailable = 0.33`.

To compound the issue, we also had no health checks on this app. That makes some conceptual sense, since it's not serving requests. Without health checks, `flyctl` considers a Machine as "deployed" once it has been running for 10 seconds without crash-looping.

The combination of these two meant that this app was deploying all of its Machines within about 60 seconds. Each Machine made a fair number of database connections at boot, and each connection caused a temporary load on our database. When you add it all up, we were overloading our primary database, meaning anything that interacted with the database (various APIs/our dashboard/our job queue workers) could be slowed/time out.

The fix here was two-pronged:
- We also reduced `max_unavailable`, but only to `0.15`, since we don't need to maintain quite as much availability for job queue workers. This was more about pacing the deploy to ease database load due to new connections rather than maintaining availability.
- We added a health check, so that the deploy couldn't rapidly blitz through all Machines. Without this health check, reducing `max_unavailable` wouldn't have done very much for us.

Since these fixes, we haven't seen any HAProxy instances mark the API as unavailable, and the number of proxy errors is significantly reduced. We'll always have some proxy errors because `rolling` doesn't cordon Machines before updating them. It's definitely got me interested in adding something like a `rolling-cordon` strategy!

The tradeoff to these changes is that our deploys are a bit slower, but certainly not unbearably so.