1 min read 1 like
An Oban Odyssey: Coda
Jacob Fenton
Last episode, I mentioned that we’d increased the Reindexer plugin timeout and made the Oban database user the owner of the oban_jobs table, so that it can actually run the reindexing query.
Unfortunately (fortunately?), we’re now merging so frequently to this app that the Reindexer plugin is being reliably interrupted by deployments whilst it is in the middle of executing. We only expect merge frequency to increase, so we need a better solution than “crossing our fingers and hoping it’ll finish reindexing tomorrow”.
That solution: crib from release_command. That is to say, Machines created viaflyctl machine run are not managed by flyctl deploy. If we tack on --rm, the Machine will self-destruct on exit. Now we’ve almost got a Machine that we can use for the reindexing.
The final catch is that we need --metadata fly_process_group=not_an_actual_process_group, because we use bluegreen strategy for deploying this app. As of flyctl v0.4.69, bluegreen refuses to deploy while unmanaged Machines belong to a process group defined in fly.toml. So we’re just setting it to something else; what it actually is isn’t hugely important, but for clarity, I used oban_reindex.
The actual command we give to the machine is bin/<release> eval MyApp.ObanReindex.run() (this is an Elixir app), which loads config without starting the app itself, so no traffic is directed to this machine (it doesn’t have services anyway, but Defense-in-Depth™ :sign_of_the_horns:).
In an effort to avoid spreading precious production deploy tokens to more places, we decided that a good home for the script that does all of this would be Buildkite. We already have prod deploy tokens in Buildkite, so there’s no new attack surface area, and Buildkite has scheduled builds which perfectly fit our use-case.
After merging the new Buildkite pipeline definition, I manually ran the job and then kicked off a production deployment concurrently. The reindexing continued unabated, like a trooper!