---
title: "First-class canary deploys make for fearless dependency updates"
description: "Canary and staging deploys now take a lease that the Buildkite pipeline enforces, so a prod deploy can no longer overwrite a running canary."
date: "2026-11-10"
author: "Jacob Fenton"
url: "https://fly.io/changelog/first-class-canary-deploys-make-for-fearless-dependency/"
discussion: "https://community.fly.io/t/first-class-canary-deploys-make-for-fearless-dependency-updates/29039"
---

# First-class canary deploys make for fearless dependency updates

Internally, for quite a while, we've had the ability to do a canary deployment of the app that serves our GraphQL API. It was, however, a slightly creaky mechanism.

## A quick aside on Fly.io's internal leasing tool

Eons ago, someone at Fly.io came up with a great idea for a Slack bot that could help to let your colleagues know if you were using a particular staging environment, and to please try again later. Basically, a small app with a (probably SQLite) database listing which staging environments were "leased", by whom, and for how long.

To take a lease is as simple as:

>**jacob** · 3:44 PM
> ```
> /lease acquire staging-env-1
> ```

> **Lisa** `APP` · 3:44 PM
> ```
> lease          user    time remaining
> -------------  ------  ----------------
> staging-env-1  jacob   59m 59s
> ```

Everyone can see you take a lock, and for how long. If anyone wants to see what environments are currently held, they just type `/lease list` in any channel and **Lisa**  (not the app's actual name, but pretty pleased with myself for coming up with this lukewarm pun on the spot) spits out the table above.

The eagle-eyed among you may notice a fatal flaw in this system: it's a pinky promise. **Lisa** isn't actually linked to our staging deployment pipeline in any way. Nothing actually stops someone else from deploying their change to `staging-env-1` whilst I hold the lock.

## The canary takes flight

An erstwhile Fly.io employee (sensibly) saw the need for canary deployments, and sought to capitalise on the ubiquity of **Lisa** amongst staff. They implemented a `--only-machines` flag for the `flyctl deploy` command.

A canary deploy could then be done merely by listing Machines for the app in question, taking the first Machine ID, and running `flyctl deploy --only-machines $MACHINE`. In order to signal your colleagues, you would also `/lease acquire canary`.

Again, this is still just the honour system. Except now, you have to check whether the canary lease is held before you merge any PR for any app with a canary capability (all PRs are automatically deployed to prod). Reader, I'll be honest: this did not happen.

The consequences were not severe - the canary would simply be updated by any subsequent deployment. This ruined any monitoring you might've been doing, but it didn't, like, break prod. But there was substantial room for improvement.

## The Vulnpocalypse

In light of the increasing frequency of dependency updates that we all, as an industry, are having to perform, we sought a more iron-clad canary deployment mechanism.

We added an authenticated API to **Lisa** for taking/releasing/listing leases. We gave the only token for that API to the Buildkite pipeline responsible for both prod and canary deployments. We changed the production deployment step to check leases before beginning - if it sees `canary` is leased, it refuses to continue. We changed the canary deploy step to ask for a duration, and then take the `canary` lease for the given duration. The canary step, like the prod step, also refuses to continue if the `canary` lease is held by someone else (we all login to Buildkite using the same email as for Slack).

Finally, we changed **Lisa** so that it sends a Slack message when someone takes the canary lease. The "until ..." uses Slack's [localised date formatting](https://docs.slack.dev/messaging/formatting-message-text/#date-formatting), a super handy feature for globally remote companies that I learnt about whilst implementing this!

> **Lisa** `APP` · 2:13 PM
>
> :locked: @jfent took `canary` for 30m, until Today at 2:43 PM, from this build.

So far, it's all working really well! We have been running lots of canaries to test dep updates recently, with an agent monitoring the canary against the rest of the fleet, looking for differences in logs/responses between the two. That process definitely would've been more shaky if we'd still been on the pinky-promise system.

We also introduced the same behaviour for staging deploys, e.g.:

> **Lisa** `APP` · 10:22 AM
>
> :locked: @jfent took `staging-env-1` for 2h, until Today at 12:22 PM, from this build.

The other nice thing about using **Lisa** here is that it still works if someone wants to take a lease the old-school way (by hand)!