Internally, for quite a while, we’ve had the ability to do a canary deployment of the app that serves our GraphQL API. It was, however, a slightly creaky mechanism.

A quick aside on Fly.io’s internal leasing tool

Eons ago, someone at Fly.io came up with a great idea for a Slack bot that could help to let your colleagues know if you were using a particular staging environment, and to please try again later. Basically, a small app with a (probably SQLite) database listing which staging environments were “leased”, by whom, and for how long.

To take a lease is as simple as:

jacob · 3:44 PM
/lease acquire staging-env-1

Lisa APP · 3:44 PM

lease          user    time remaining
-------------  ------  ----------------
staging-env-1  jacob   59m 59s

Everyone can see you take a lock, and for how long. If anyone wants to see what environments are currently held, they just type /lease list in any channel and Lisa (not the app’s actual name, but pretty pleased with myself for coming up with this lukewarm pun on the spot) spits out the table above.

The eagle-eyed among you may notice a fatal flaw in this system: it’s a pinky promise. Lisa isn’t actually linked to our staging deployment pipeline in any way. Nothing actually stops someone else from deploying their change to staging-env-1 whilst I hold the lock.

The canary takes flight

An erstwhile Fly.io employee (sensibly) saw the need for canary deployments, and sought to capitalise on the ubiquity of Lisa amongst staff. They implemented a --only-machines flag for the flyctl deploy command.

A canary deploy could then be done merely by listing Machines for the app in question, taking the first Machine ID, and running flyctl deploy --only-machines $MACHINE. In order to signal your colleagues, you would also /lease acquire canary.

Again, this is still just the honour system. Except now, you have to check whether the canary lease is held before you merge any PR for any app with a canary capability (all PRs are automatically deployed to prod). Reader, I’ll be honest: this did not happen.

The consequences were not severe - the canary would simply be updated by any subsequent deployment. This ruined any monitoring you might’ve been doing, but it didn’t, like, break prod. But there was substantial room for improvement.

The Vulnpocalypse

In light of the increasing frequency of dependency updates that we all, as an industry, are having to perform, we sought a more iron-clad canary deployment mechanism.

We added an authenticated API to Lisa for taking/releasing/listing leases. We gave the only token for that API to the Buildkite pipeline responsible for both prod and canary deployments. We changed the production deployment step to check leases before beginning - if it sees canary is leased, it refuses to continue. We changed the canary deploy step to ask for a duration, and then take the canary lease for the given duration. The canary step, like the prod step, also refuses to continue if the canary lease is held by someone else (we all login to Buildkite using the same email as for Slack).

Finally, we changed Lisa so that it sends a Slack message when someone takes the canary lease. The “until …” uses Slack’s localised date formatting, a super handy feature for globally remote companies that I learnt about whilst implementing this!

Lisa APP · 2:13 PM

:locked: @jfent took canary for 30m, until Today at 2:43 PM, from this build.

So far, it’s all working really well! We have been running lots of canaries to test dep updates recently, with an agent monitoring the canary against the rest of the fleet, looking for differences in logs/responses between the two. That process definitely would’ve been more shaky if we’d still been on the pinky-promise system.

We also introduced the same behaviour for staging deploys, e.g.:

Lisa APP · 10:22 AM

:locked: @jfent took staging-env-1 for 2h, until Today at 12:22 PM, from this build.

The other nice thing about using Lisa here is that it still works if someone wants to take a lease the old-school way (by hand)!