---
title: "Background Workers: Running Web, Worker, Redis and Cron Together"
description: "What background workers are, which queue library to use, and how to deploy a web app, worker, Redis and cron from one image without running jobs twice."
---

# Background Workers: Running Web, Worker, Redis and Cron Together

![Daniel Botha](https://fly.io/static/images/danielb.webp)

Daniel Botha

Writer at Fly.io

Published

Oct 22, 2026

A background worker is a long-running process that sits outside your web server’s request cycle, pulls jobs off a queue, and runs them, so slow or failure-prone work never holds an HTTP request open.

A user signs up. Your app needs to send a welcome email, resize the avatar they uploaded, and push the account to your CRM. Done inline, that request waits on three outside services, and when the email provider is slow it times out and the user clicks the button again. So the request handler drops a job on a queue and returns immediately, and a separate process does the work.

That separate process is easy to write and easy to get wrong in production. It needs a broker both processes can reach, a deploy that ships the same code to both, a way to stop without dropping jobs, and some answer for the nightly cleanup job that has to run exactly once.

---

## What Is a Background Worker?

A background worker is a process that runs continuously, receives no incoming network traffic, and processes jobs that other parts of the system put on a queue.

### The three moving parts

Every worker setup has the same three pieces. The **producer** is your web app: a request handler or model callback that serializes a job (a function name and its arguments) and pushes it onto the queue. The **broker** stores pending jobs until something takes them; Celery, for example, supports Redis, RabbitMQ and Amazon SQS as stable brokers.[1] The **worker** is the consumer: it blocks on the queue, takes a job, runs it, and marks it done or failed.

Most jobs are fire-and-forget (send this email), some are delayed (a reminder 24 hours after signup), and some are recurring (the nightly report). Recurring jobs need a scheduler as well as a worker, which is where most of the trouble starts.

---

## Should Background Work Run in the Same Process as the Web Server?

No, not for work that matters. An in-process task works until the server restarts, at which point any job it was holding is gone with no record it existed, and a CPU-heavy job competes with requests for the same cores and memory.

| layout | how it runs | what goes wrong |
| --- | --- | --- |
| In-process task | A thread or async task inside the web server | Jobs vanish on restart; heavy jobs slow requests |
| Separate process, same box | Web and worker started side by side by a process manager | One box’s memory is shared, so a runaway job can take the site down with it |
| Separate process, own machine | Web and worker built from one codebase, run as separate processes on separate machines | Needs a deploy that knows about both |

Run the third layout: the same build, started with different commands, each process with its own memory and its own count. What to avoid is a worker built as a separate project with its own deploy pipeline. Job classes and settings drift apart, and a job enqueued by new web code gets picked up by an old worker that does not know its arguments.

---

## Which Queue Library Should I Use?

Use the one your framework’s community already runs: Celery for Python, Sidekiq or Solid Queue for Rails, BullMQ for Node.js, and Laravel’s built-in queue for PHP.

| language | library | broker | worker command | recurring jobs |
| --- | --- | --- | --- | --- |
| Python | Celery | Redis, RabbitMQ, SQS | `celery -A proj worker -l INFO` | Celery Beat, a separate process |
| Ruby | Sidekiq | Redis | `bundle exec sidekiq` | Periodic jobs in Sidekiq Enterprise |
| Ruby | Solid Queue | Your SQL database | `bin/jobs` | `config/recurring.yml` |
| Node.js | BullMQ | Redis | Your script that creates a `Worker` | `upsertJobScheduler` with a cron pattern |
| PHP | Laravel queues | Redis or database | `php artisan queue:work` | `php artisan schedule:run` every minute |

*Sources: [2][1][3][4][5][6][7][8][9][10]. Checked 2026-10-22.*

### Redis or your database as the queue

Sidekiq reads its Redis connection from the `REDIS_URL` environment variable and falls back to `localhost:6379` when that is unset, which is why a worker that “works locally” often fails on first deploy.[11]

A database-backed queue removes the broker entirely. Solid Queue is a database-backed Active Job backend and ships preconfigured in new Rails 8 apps,[7] and Laravel’s worker can run against the `database` connection.[9] The tradeoff arrives at high throughput, when the queue’s polling load lands on the database serving your requests.

### Write jobs to run twice

Whatever library you pick, assume a job can run more than once. Sidekiq’s guidance is that it executes a job at least once, not exactly once, so jobs should be idempotent.[12] A worker killed mid-job produces a second run, so check whether the email was already sent before sending it.

---

## How Do I Deploy a Web App, Worker, Redis and Cron Together?

Build one image, start it with different commands for web and worker, point both at the same broker and database through environment variables, and run exactly one scheduler.

The web server and the worker should come from the same commit and be updated by the same deploy, so a job enqueued by new code is processed by new code. Both need the same `DATABASE_URL` and `REDIS_URL`, set once as application secrets rather than per process, so they cannot diverge.

### Stopping without losing work

Every deploy stops the old worker, and what happens to the job in hand depends on the signal and the time it is given. Celery treats `TERM` as a warm shutdown and finishes its currently executing tasks before exiting.[2] BullMQ’s `worker.close()` stops the worker picking up new jobs and waits for the current ones to finish.[13] Laravel’s docs warn that queue workers are long-lived processes that will not notice code changes until they are restarted, which is why a Laravel deploy on a long-lived server needs a `queue:restart` step.[9]

---

## How Should Scheduled Jobs (Cron) Run Next to Workers?

Run exactly one scheduler, and have it enqueue jobs for the workers rather than doing heavy work itself. The common failure is running the scheduler inside every web or worker replica, so that scaling to three machines sends the nightly report three times.

Celery allows only a single beat scheduler per schedule, and says embedding beat inside a worker with the `-B` flag is not recommended for production.[3] Laravel’s scheduler expects a single cron entry calling `php artisan schedule:run` every minute, and offers `onOneServer()` with a shared cache lock for when the scheduler runs on several servers.[10] BullMQ’s job schedulers are stored in Redis, so the schedule lives with the queue rather than in a process.[8]

That leaves two shapes. A **scheduler process** (Celery Beat, or a cron daemon running your crontab) runs with a count of one, fires on precise cron expressions and enqueues jobs. A **scheduled one-off run** starts a short-lived process from your image on an interval, runs one command and stops, which suits housekeeping like a nightly cleanup.

---

## Background Workers on Fly.io

On Fly.io, a web server, a worker and a scheduler are process groups in one `fly.toml`. They come from one image and ship in one `fly deploy`; there are no separate services to wire together. `fly launch` detects the framework and writes the Dockerfile and `fly.toml`; adding a worker is a `[processes]` block.

Each process group runs on its own Fly Machines, so the worker does not share memory or CPU with the web server, and `fly deploy` creates at least one Machine for every group it finds. Here is a Django app with Celery and Celery Beat, each group sized on its own:

```
app = "my-app"
primary_region = "iad"
kill_signal = "SIGTERM"
kill_timeout = 120

[processes]
  web = "gunicorn myproject.wsgi --bind 0.0.0.0:8000"
  worker = "celery -A myproject worker -l INFO"
  beat = "celery -A myproject beat -l INFO"

[deploy]
  release_command = "python manage.py migrate"

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = "stop"
  auto_start_machines = true
  min_machines_running = 1
  processes = ["web"]

[[vm]]
  processes = ["web", "worker"]
  memory = "1gb"

[[vm]]
  processes = ["beat"]
  memory = "256mb"
```

Only `web` is attached to the HTTP service. Auto stop is a setting on a service, so the worker and beat Machines, which have none, receive no traffic and keep running. `kill_signal = "SIGTERM"` triggers Celery’s warm shutdown, and `kill_timeout` gives a running task up to 120 seconds to finish; the maximum is 300. Because `fly deploy` moves every Machine in every group onto the new release, the worker comes back on the new code without a separate restart step.

For other frameworks, only the `[processes]` lines change:

| framework | `worker` | scheduler |
| --- | --- | --- |
| Rails + Sidekiq | `bundle exec sidekiq` | Sidekiq Enterprise periodic jobs, or `cron = "supercronic /app/crontab"` |
| Rails + Solid Queue | `bin/jobs` | `config/recurring.yml`, run by `bin/jobs` |
| Node.js + BullMQ | `node worker.js` | `upsertJobScheduler` in your app code |
| Laravel | `php artisan queue:work redis` | `cron = "supercronic /app/crontab"`, with supercronic in the image and a `schedule:run` line in the crontab |

The database and broker attach as secrets both processes read. `fly mpg create` provisions [Managed Postgres](https://fly.io/learn/what-is-the-best-postgres-hosting/) and `fly mpg attach <cluster-id> -a my-app` sets `DATABASE_URL` on the app. `fly redis create` provisions [Upstash for Redis](https://fly.io/docs/upstash/redis/), a Redis-compatible service from Upstash, provisioned in your Fly.io organization and reachable from your apps at a private address; copy the Private URL from `fly redis status` and set it with `fly secrets set REDIS_URL=...`.

Scale each group on its own count. `fly scale count web=2 worker=4` adds workers without touching the web tier, and the [process groups docs](https://fly.io/docs/launch/processes/) cover the rest. Keep `beat` or `cron` at a count of one.

For jobs that only need to run roughly once an hour, day, week or month, a scheduled Machine replaces the scheduler process:

```
fly machine run . python manage.py clearsessions --schedule daily
```

The `.` builds the Machine from your Dockerfile, and like every Machine in the app it gets the app’s secrets, so `DATABASE_URL` is there. It starts on a fuzzy interval, runs the command and stops. Fly.io’s [task scheduling guide](https://fly.io/docs/blueprints/task-scheduling/) says plainly that scheduled Machines are not for precise timing; for “every day at 03:15”, run a scheduler process instead. Framework guides for [Django](https://fly.io/learn/django-hosting/), [Rails](https://fly.io/learn/rails-hosting/) and [Laravel](https://fly.io/learn/laravel-hosting/) cover the rest of each stack, and [Fly.io vs Render](https://fly.io/learn/fly-vs-render/) compares the two platforms.

Priced from Fly.io’s published rates for `iad`, the stack above costs about $66.39 a month before bandwidth: $6.70 each for the 1 GB shared-cpu-1x web and worker Machines, $2.19 for the 256 MB beat Machine, $38 for the Managed Postgres Basic plan with high availability and backups, $2.80 for 10 GB of database storage, and $10 for Upstash’s Fixed 250MB plan.[14][15][16] Other sizes and regions can be priced at [fly.io/calculator](https://fly.io/calculator/).

---

## Frequently Asked Questions

### Do background workers need their own server?

No, but they need their own process, and in production that process usually runs on its own machine so a heavy job cannot starve the web server. On Fly.io the worker is a process group in the same fly.toml as the web server, so it runs on its own Fly Machines while sharing one image and one deploy.

### Can I use Postgres instead of Redis as a job queue?

Yes. Solid Queue for Rails and Laravel’s database queue driver both store jobs in your SQL database, which removes Redis from the stack. On Fly.io that database can be Managed Postgres, attached with fly mpg attach. At high job volume the queue’s polling load lands on the same database that serves requests, so watch it as you grow.

### How do I scale background workers independently of the web server?

Run the worker as its own process with its own instance count. On Fly.io each process group in fly.toml scales separately, so fly scale count worker=4 adds workers without changing the number of web Machines, and a [[vm]] section can give workers more memory than the web tier.

### Does a worker process need a port?

No. A worker pulls jobs from the queue and receives no incoming traffic, so it needs no public port. On Fly.io, attach only the web process group to the http_service and the worker Machines run without a service, so auto stop, which is a service setting, never applies to them.

### How do I stop a cron job running twice when I scale up?

Run exactly one scheduler. Celery requires a single beat process per schedule, and Laravel offers onOneServer for schedulers on several servers. On Fly.io, put the scheduler in its own process group and keep it at a count of one, or use a scheduled Machine for coarse jobs.

### What happens to a job when a worker restarts or deploys?

It depends on the shutdown signal and how long the worker is given. Celery finishes running tasks on SIGTERM and BullMQ waits for current jobs on close. On Fly.io, set kill *signal to SIGTERM and kill* timeout up to 300 seconds, and write jobs to be safe if they run twice.

### How do I run Celery Beat or the Laravel scheduler on Fly.io?

Add it as its own process group with a count of one. For Celery that is a beat process running celery beat. For Laravel it is a cron process running supercronic with a crontab that calls php artisan schedule:run every minute. Both ship in the same fly deploy as the web server.

### How do I see worker logs on Fly.io?

Fly.io collects stdout and stderr from every Machine in the app. Run fly logs to stream it; each line carries the ID of the Machine it came from, and fly logs –machine with a worker Machine’s ID shows that worker alone.

---

## Sources

1. ^ Celery Project, “Backends and Brokers”. [docs.celeryq.dev/en/stable/getting-started/backends-and-brokers](https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/index.html). Checked 2026-10-22.
2. ^ Celery Project, “Workers Guide”. [docs.celeryq.dev/en/stable/userguide/workers.html](https://docs.celeryq.dev/en/stable/userguide/workers.html). Checked 2026-10-22.
3. ^ Celery Project, “Periodic Tasks”. [docs.celeryq.dev/en/stable/userguide/periodic-tasks.html](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html). Checked 2026-10-22.
4. ^ Sidekiq, “Getting Started”. [github.com/sidekiq/sidekiq/wiki/Getting-Started](https://github.com/sidekiq/sidekiq/wiki/Getting-Started). Checked 2026-10-22.
5. ^ Sidekiq, “Ent Periodic Jobs”. [github.com/sidekiq/sidekiq/wiki/Ent-Periodic-Jobs](https://github.com/sidekiq/sidekiq/wiki/Ent-Periodic-Jobs). Checked 2026-10-22.
6. ^ BullMQ, “Workers”. [docs.bullmq.io/guide/workers](https://docs.bullmq.io/guide/workers). Checked 2026-10-22.
7. ^ Rails, “Solid Queue”. [github.com/rails/solid_queue](https://github.com/rails/solid_queue). Checked 2026-10-22.
8. ^ BullMQ, “Job Schedulers”. [docs.bullmq.io/guide/job-schedulers](https://docs.bullmq.io/guide/job-schedulers). Checked 2026-10-22.
9. ^ Laravel, “Queues”. [laravel.com/docs/12.x/queues](https://laravel.com/docs/12.x/queues). Checked 2026-10-22.
10. ^ Laravel, “Task Scheduling”. [laravel.com/docs/12.x/scheduling](https://laravel.com/docs/12.x/scheduling). Checked 2026-10-22.
11. ^ Sidekiq, “Using Redis”. [github.com/sidekiq/sidekiq/wiki/Using-Redis](https://github.com/sidekiq/sidekiq/wiki/Using-Redis). Checked 2026-10-22.
12. ^ Sidekiq, “Best Practices”. [github.com/sidekiq/sidekiq/wiki/Best-Practices](https://github.com/sidekiq/sidekiq/wiki/Best-Practices). Checked 2026-10-22.
13. ^ BullMQ, “Graceful shutdown”. [docs.bullmq.io/guide/workers/graceful-shutdown](https://docs.bullmq.io/guide/workers/graceful-shutdown). Checked 2026-10-22.
14. ^ Fly.io, “Fly.io Resource Pricing”. [fly.io/docs/about/pricing](https://fly.io/docs/about/pricing/). Checked 2026-10-22.
15. ^ Fly.io, “Managed Postgres”. [fly.io/docs/mpg](https://fly.io/docs/mpg/). Checked 2026-10-22.
16. ^ Fly.io, “Upstash for Redis”. [fly.io/docs/upstash/redis](https://fly.io/docs/upstash/redis/). Checked 2026-10-22.

## Ship web and workers

Run your web server, workers and scheduler from one fly.toml and one deploy.

[Sign up](https://fly.io/app/sign-up)