Sidekiq Background Workers
Rails applications commonly defer complex tasks that take a long to complete to a background worker to make web responses seem fast. This guide shows how to use Sidekiq, a popular open-source Rails background job framework, to set up background workers, but it could be done with other great libraries like Good Job, Resque, etc.
Provision a Redis Server
Sidekiq depends on Redis to communicate between the Rails server process and the background workers. Follow the Redis setup guide to provision a Redis server and set a REDIS_URL
within the Rails app. Be sure to set the REDIS_URL
via a secret as demonstrated here.
Verify the REDIS_URL
is available to your Rails application before you continue by running:
fly ssh console -C "printenv REDIS_URL"
REDIS_URL=redis://default:yoursecretpassword@my-apps-redis-host.internal:6379
If you don't see REDIS_URL
in the command above, Sidekiq won't be able to connect and process background jobs.
Run Multiple Processes
Most production Rails applications run background workers in a separate process. There's a few ways of accomplishing that on Fly that are outlined in the multiple-processes docs.
The quickest way to run multiple processes in one region is via the processes
directive in the fly.toml
file.
Add the following to the fly.toml
:
[processes]
web = "bin/rails fly:server"
worker = "bundle exec sidekiq"
Then under the [[services]]
directive, find the entry that maps to internal_port = 8080
, and change processes = ["app"]
to processes = ["web"]
. The configuration file should look something like this:
[[services]]
processes = ["web"] # this service only applies to the web process
http_checks = []
internal_port = 8080
protocol = "tcp"
script_checks = []
This associates the process with the service that Fly launches.
Deploy and Test
Once multiple processes are configured in the fly.toml
file, deploy them via:
fly deploy
If all goes well the application should launch with both web
and worker
processes. Be sure to run through the application and test features that kick-off background jobs. If you're having issues getting it working, run fly logs
to see errors.
Scaling
Scaling up and down processes may be accomplished by running:
fly scale count web=3 worker=3
To view the current state of the application's scale, run:
fly status
App
Name = my-rails-app
Owner = personal
Version = 41
Status = running
Hostname = my-rails-app.fly.dev
Instances
ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED
15088508 worker 41 ord run running 0 34s ago
8789ef49 web 41 ord run running 1 total, 1 passing 0 2022-07-26T16:06:34Z
c419942b web 41 ord run running 1 total, 1 passing 0 2022-07-26T16:05:52Z
ea7af986 web 41 ord run running 1 total, 1 passing 0 2022-07-26T16:05:52Z
d681c33d worker 41 ord run running 0 2022-07-26T15:42:30Z
d8d8dc08 worker 41 ord run running 0 2022-07-26T15:42:30Z
In this case, we can see that 3 worker processes and 3 web processes are running in the ord
region.