Sidekiq Background Workers
Rails applications commonly defer complex tasks that take a long to complete to a background worker to make web responses seem fast. This guide shows how to use Sidekiq, a popular open-source Rails background job framework, to set up background workers, but it could be done with other great libraries like Good Job, Resque, etc.
Provision a Redis Server
Sidekiq depends on Redis to communicate between the Rails server process and the background workers. Follow the Redis setup guide to provision a Redis server and set a
REDIS_URL within the Rails app. Be sure to set the
REDIS_URL via a secret as demonstrated here.
REDIS_URL is available to your Rails application before you continue by running:
fly ssh console -C "printenv REDIS_URL"
If you don't see
REDIS_URL in the command above, Sidekiq won't be able to connect and process background jobs.
Run Multiple Processes
Most production Rails applications run background workers in a separate process. There's a few ways of accomplishing that on Fly that are outlined in the multiple-processes docs.
The quickest way to run multiple processes in one region is via the
processes directive in the
Add the following to the
[processes] app = "bin/rails server" worker = "bundle exec sidekiq"
Then under the
[http_service] directive, add
processes = ["app"]. The configuration file should look something like this:
[http_service] processes = ["app"] # this service only applies to the app process internal_port = 3000 force_https = true auto_stop_machines = true auto_start_machines = true min_machines_running = 0
This associates the process with the service that Fly launches.
Deploy and Test
Once multiple processes are configured in the
fly.toml file, deploy them via:
If all goes well the application should launch with both
worker processes. Be sure to run through the application and test features that kick-off background jobs. If you're having issues getting it working, run
fly logs to see errors.
Scaling up and down processes may be accomplished by running:
fly scale count app=3 worker=3
To view the current state of the application's scale, run:
App Name = my-rails-app Owner = personal Version = 41 Status = running Hostname = my-rails-app.fly.dev Instances ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED 15088508 worker 41 ord run running 0 34s ago 8789ef49 app 41 ord run running 1 total, 1 passing 0 2022-07-26T16:06:34Z c419942b app 41 ord run running 1 total, 1 passing 0 2022-07-26T16:05:52Z ea7af986 app 41 ord run running 1 total, 1 passing 0 2022-07-26T16:05:52Z d681c33d worker 41 ord run running 0 2022-07-26T15:42:30Z d8d8dc08 worker 41 ord run running 0 2022-07-26T15:42:30Z
In this case, we can see that 3 worker processes and 3 app processes are running in the