Django Beats

ASGI deployment options for Django

2024-03-27T00:00:00+00:00

Mariusz Felisiak, a Django and Python contributor and a Django Fellow, explores the world of ASGI servers. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

Asynchronous support in Django is constantly improving and each version introduces new features. With already implemented:

asynchronous views and middlewares,
asynchronous ORM interface,
asynchronous support for decorators and signal dispatchers,

and growing support in contrib packages (like auth or sessions), it’s now possible to create a fully-asynchronous real world application in Django. You can find more details on this topic in one of my previous blog posts. To efficiently serve an asynchronous app and avoid unnecessary context-switching that causes a performance penalty, we need an ASGI server (Asynchronous Server Gateway Interface). This article explores the most popular ASGI web servers and how to integrate them in our development and deployment processes.

Let’s start with a brief introduction of the ASGI application.

ASGI application

The application object is used by the ASGI-compatible web servers to communicate with your project. ASGI application is defined in the asgi.py file generated by the startproject management command in the project root directory. This works analogously to the WSGI application defined for WSGI-compatible web servers in the wsgi.py. This is what the hello_django project structure looks like with two entry-points for web servers:

hello-django/
    manage.py
    hello_django/
        __init__.py
        settings.py
        urls.py
        asgi.py  ← Entry-point for ASGI-compatible web servers.
        wsgi.py  ← Entry-point for WSGI-compatible web servers.

Most ASGI web servers accept path to the application callable, so we will pass it as: <project name>.asgi:application, e.g. hello_django.asgi:application.

Now let’s dive into the most popular ASGI web servers 🤿

Daphne

Daphne is a pure-Python ASGI server that supports HTTP/1, HTTP/2, and WebSockets. It’s written and maintained under the wing of the Django organization which is a big plus from a long-term support perspective. It should be maintained as long as Django exists.

Another advantage is that it provides integration with the built-in runserver command, which allows you to run your project under ASGI during development and benefit from all runserver functionalities, such as system checks, auto-reloader, or readable logs.

To enable runserver integration, add daphne to the INSTALLED_APPS and define ASGI_APPLICATION with the path to your ASGI application in your settings file. For example:

    # hello_world/settings.py

INSTALLED_APPS = [
    "daphne",  # ← Added.
    ...,
]

ASGI_APPLICATION = "hello_django.asgi.application"  # ← Added.

  

Now runserver will use Daphne development server:

python manage.py runserver

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

March 19, 2024 - 10:18:47
Django version 5.0.3, using settings 'hello_django.settings'
Starting ASGI/Daphne version 4.1.0 development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

To deploy on ASGI using Daphne, first make sure that it’s installed and listed in requirements.txt:

python -m pip install Daphne

pip freeze > requirements.txt

Second, update Dockerfile generated by fly launch to use Daphne instead of Gunicorn:

    # Dockerfile
...

EXPOSE 8000

# ↓↓ Update to Daphne ↓↓
CMD ["daphne", "-b", "0.0.0.0", "-p", "8000", "hello_django.asgi:application"]

Hypercorn

Hypercorn is a pure-Python hybrid ASGI/WSGI server that supports HTTP/1, HTTP/2, HTTP/3 (optionally using aioquic library), and WebSockets. It’s really flexible and can be used both during development and as a production web server. It also has a medley of configuration options, including:

--worker-class: the type of worker to use, with support for various worker classes:
- asyncio - default
- uvloop - available when installed with hypercorn[uvloop]
- trio - available when installed with hypercorn[trio]
--reload: enables auto-reload on code changes, which is very handy for local development.

To deploy on ASGI using Hypercorn, first make sure that it’s installed and listed in requirements.txt:

python -m pip install hypercorn

pip freeze > requirements.txt

Second, update Dockerfile generated by fly launch to use Hypercorn instead of Gunicorn:

    # Dockerfile
...

EXPOSE 8000

# ↓↓ Update to Hypercorn ↓↓
CMD ["hypercorn", "--bind", ":8000", "--workers", "2", "hello_django.asgi:application"]

Uvicorn

Uvicorn is a pure-Python ASGI server based on uvloop that supports HTTP/1.1 and WebSockets. Use the uvicorn command to run your project under ASGI during development:

uvicorn hello_django.asgi:application

INFO:     Started server process [6591]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

Passing --reload causes the server to reload when Python files are changed. We can pass --reload-include to extend this behavior for other files, e.g. HTML templates:

uvicorn --reload --reload-include *.py --reload-include *.html hello_django.asgi:application

INFO:     Will watch for changes in these directories: ['/examples/hello_django']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [6682] using WatchFiles
INFO:     Started server process [6684]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

To deploy on ASGI using Uvicorn, first make sure that both Uvicorn and Gunicorn are installed and listed in requirements.txt:

python -m pip install uvicorn gunicorn

pip freeze > requirements.txt

Second, update Dockerfile generated by fly launch to start Gunicorn using the Uvicorn worker class:

    # Dockerfile
...

EXPOSE 8000

# ↓↓ Update to the Uvicorn worker class ↓↓
CMD ["gunicorn", "--bind", ":8000", "--workers", "2", "-k", "uvicorn.workers.UvicornWorker", "hello_django.asgi:application"]

Gunicorn is a combat-proven web server that implements many essential features, that’s why it’s recommended for running Uvicorn workers in a production environment.

Granian

Granian is a newish option in ASGI servers world. It is a Rust 🦀 hybrid ASGI/WSGI/RSGI server that supports HTTP/1, HTTP/2, and WebSockets. It can be used both during development and as a production web server. It has a myriad of configuration options and provides an auto-reloader (when installed with granian[reload]) that is useful for development. Use the granian command to develop your project under ASGI:

granian --reload hello_django.asgi:application

[INFO] Starting granian (main PID: 6049)
[INFO] Listening at: 127.0.0.1:8000
[INFO] Spawning worker-1 with pid: 6050
[INFO] Started worker-1
[INFO] Started worker-1 runtime-1

The only thing I personally miss is the lack of access logs, but this may change in the future.

To deploy on ASGI using Granian, first make sure that it’s installed and listed in requirements.txt:

python -m pip install granian

pip freeze > requirements.txt

Second, update Dockerfile generated by fly launch to use Granian instead of Gunicorn:

    # Dockerfile
...

EXPOSE 8000

# ↓↓ Update to Granian ↓↓
CMD ["granian", "--interface", "asgi", "--host", "0.0.0.0", "--port", "8000", "--workers", "2", "hello_django.asgi:application"]

Closing thoughts

The world of ASGI servers is truly rich and everyone should find something that suits their personal preferences. This article mentions just some of the most popular and solid options. Choosing an ASGI server for a production environment is like picking an ice cream flavor - everyone has to find their favorite type based on personal taste.

Try them and share!

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to deploy those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

Shedding light on Django versioning

2024-01-30T00:00:00+00:00

Mariusz Felisiak, a Django and Python contributor and a Django Fellow, sheds light on Django versioning and release cycle. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

As of version 2.0 released in 2017, Django uses a loose form of semantic versioning, but what exactly does that mean? This article dives into the details of Django’s release cycle and related policies to answer some on the crucial questions, such as:

What we can expect from each version in the release series?
Does Django break things? if so, when?
How long is each version supported?
What’s Django policy for supporting Python versions?

Let’s start with a brief description on each release series.

Release cycle

Django releases come in series, and each series contains three releases. The leading numbers of each series are consecutive and have no special meaning. Django contributors work now on 5.X series, the next one will be 6.X, and so on. The current series (like any other in the future) will contain three releases:

5.0 (released on December 4, 2023)
5.1 (planned for August, 2024)
5.2 - Long Term Support (planned for April 2025)

A new feature release (X.Y) is issued every 8 months. It is preceded by almost 3 months of pre-releases (alpha, beta, release candidate) before the final version is released. So please don’t hesitate and test pre-releases to help bulletproofing the final version 🐛

Moreover, you should be aware that each X.Y release brings tons of new features and smaller improvements, so do not expect that e.g. 5.0 will differ significantly from 5.1 or 5.2 in this matter. Each new version is the best Django ever!

After release, every version (including LTS) is in the mainstream support for exactly 8 months, what means that a minor version (X.Y.Z) containing bugfixes will be issued for them at the beginning of each month. Django has a strict backporting policy so only specific bugfixes are backported to already released versions, which includes:

security patches,
data loss and crashing bugs,
bugs in the new features,
regressions from older versions introduced in the current release series.

Once a release reaches the end of mainstream support it enters into extended support for another 8 months for a normal release (X.0 and X.1), and 28 months - for an LTS (X.2). That’s right, the total support period for the LTS versions is 3 years! Please note that only security patches and data loss bugs are fixed during the extended support.

For example, a regression introduced in Django 4.2 that was reported in January, 2024 (so when it’s already in extended support) will be only fixed in Django 5.1+ 🙀 This policy is really strict and can be scary at the beginning, but it helps protect Django and makes bumping versions really smooth. This is also why it’s so important to upgrade your Django before it reaches the end of the mainstream support. Django rewards early adopters!

Let’s note that the mainstream support for one version overlaps with the development period of the next one, both last 8 months. As a result, only one version of Django is mainstream supported at any given time. It’s visualized nicely on the Django’s download page.

When we choose our tech stack we need to answer a few key questions. Is it mature and stable? Does this break things? If so, will we have time to prepare for it? The next section answers these questions.

Stability

Django is truly committed to its stability policy, so much so that it becomes one of its main advantages. Anything that is considered part of the public API cannot be changed without appropriate deprecation, which includes clear documentation, existing upgrade paths, and cannot be done undetected by raising deprecation warnings. Let’s take a look at key points of Django policy:

deprecated features will continue to work in a backward compatible manner throughout the series in which they were deprecated,
deprecated features are never removed in LTS releases,
deprecation is done over at least 2 feature releases.

How it works in practice?

Features deprecated in 5.0 will be removed in 6.0 (2 years later).
Features deprecated in 5.1 will be removed in 6.0 (16 months later).
Features deprecated in 5.2 (LTS) will be removed in 6.1 (16 months later).

It’s heartwarming that we have plenty of time to deal with potentially breaking changes.

Django is a Python web framework, so it requires Python and it loves Python. Django is also very generous when it comes to the variety of Python versions that are supported by each release. Let’s check it in detail.

Python support

Django is one of the early adopters of new versions of Python. It’s checked daily against all latest pre-releases (starting from the very first alpha of each Python version). This approach allows Django to add official support for the latest Python as soon as it’s released. Moreover, it’s added not only to the latest Django version, but to all versions under mainstream support, e.g. support for Python 3.12 was added to Django 4.2 in version 4.2.8, so the last bugfix release for its mainstream support.

What about dropping support for old Python versions? Django is really careful here, proving once again that it cares about developers and takes stability seriously. Support for old Python versions is dropped only in the first release of each series (e.g. 5.0) which has two big advantages:

we can upgrade Django to the newest version in the same series without struggling with a Python upgrade, as all of them (X.0, X.1, X2 - LTS) will support (at least) all Python versions supported by the first one (X.0). For example:

Django version	Python versions
4.0	3.8, 3.9, 3.10
4.1	3.8, 3.9, 3.10, 3.11 (added in 4.1.3)
4.2	3.8, 3.9, 3.10, 3.11, 3.12 (added in 4.2.8)

if upgrading the Python version is somehow problematic for us, we can still use a previous LTS (X.2) version, and have 2+ more years of extended support to handle it.

Closing thoughts

Django versioning can be tricky to understand at first glance. However, when we discover and understand the reasons behind it, it becomes clear why it looks that way. Creating releases in clear and predictable series, makes life easier for developers and proves that Django really cares about them. Stability is one the greatest advantages of Django, making it the obvious choice for web developers for many years.

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to deploy those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

Django Views as Serverless Functions on Fly Machines

2024-01-10T00:00:00+00:00

In this post, we demonstrate how to use Fly Machines to run Django Views as Serverless Functions. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

In my last post, we ran custom Django commands as serverless functions by defining a new process to run within a Machine. This article continues our series transforming our Django views into serverless functions by defining a service.

But first, let’s understand some concepts about Fly Machines.

Fly Machines `processes`

When running a custom Django command, we define the command to run as a separate process in processes within a Machine:

    import requests

FLY_API_HOSTNAME = "https://api.machines.dev"
FLY_API_TOKEN = os.environ.get("FLY_API_TOKEN")
FLY_APP_NAME = os.environ.get("FLY_APP_NAME")
FLY_IMAGE_REF = os.environ.get("FLY_IMAGE_REF")

headers = {
    "Authorization": f"Bearer {FLY_API_TOKEN}",
    "Content-Type": "application/json",
}

def create_process_machine(cmd):
    config = {
        "config": {
            "image": FLY_IMAGE_REF,
            "auto_destroy": True,
            "processes": [  # ← processes here!
                {
                    "cmd": cmd.split(" ")
                }
            ],
        }
    }
    url = f"{FLY_API_HOSTNAME}/v1/apps/{FLY_APP_NAME}/machines"
    response = requests.post(
        url, headers=headers, json=config
    )
    if response.ok:
        return response.json()
    else:
        response.raise_for_status()

  

The Django command (cmd) is passed as an argument to the entrypoint of our Docker container. In this case, the Machine will stop when the process exits without an error. When that happens, the Machine will destroy itself once it’s complete, as per "auto_destroy": True. This works well for Django commands, that can be run as its own process and exits when the command finishes.

On the other side, as mention before, when creating/destroying machines we should take the boot time into consideration. An interesting alternative is to reuse existing Machines by start/stop them accordingly. That’s what we use in this post: create a machine to offload some work, stop it when it’s idle and start it when a new request comes in. For that to happen automatically, we are setting up a service.

Fly Machines `services`

By default, Machines are closed to the public internet. To make them accessible, we need to allocate an IP address to the Fly App and add one or more services.

This happens automatically when we fly launch a Django app on Fly.io and the fly.toml config file is generated with:

    # fly.toml
...
[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ["app"]

  

[http_service] is a simplified version of [[services]] section for apps that only need HTTP and HTTPS services.

In services, we specify how to expose our application to the outside world which include setting for routing incoming requests to the appropriate components of our application.

To create a Machine with a service via Machines API, we would have something like this:

    import os

import requests

FLY_API_HOSTNAME = "https://api.machines.dev"
FLY_API_TOKEN = os.environ.get("FLY_API_TOKEN")
FLY_APP_NAME = os.environ.get("FLY_APP_NAME")
FLY_IMAGE_REF = os.environ.get("FLY_IMAGE_REF")

headers = {
    "Authorization": f"Bearer {FLY_API_TOKEN}",
    "Content-Type": "application/json",
}

def create_service_machine():
    config = {
        "config": {
            "image": FLY_IMAGE_REF,
            "services": [  # ← services here!
                {
                    "protocol": "tcp",
                    "internal_port": 8000,
                    # Fly Proxy stops Machines when 
                    # this service goes idle
                    "autostop": True,
                    "ports": [
                        {
                            "handlers": ["tls", "http"],
                            # The internet-exposed port 
                            # to receive traffic on
                            "port": 443,
                        },
                        {
                            "handlers": ["http"],
                            # The internet-exposed port 
                            # to receive traffic on
                            "port": 80
                        },
                    ],
                }
            ],
        }
    }
    url = (
        f"{FLY_API_HOSTNAME}/v1/apps/{FLY_APP_NAME}/machines"
    )
    response = requests.post(
        url, headers=headers, json=config
    )
    if response.ok:
        return response.json()
    else:
        response.raise_for_status()

  

create_service_machine() creates a new machine with public access - we might refer to this machine as the serverless machine along this article.

That’s awesome but as I mentioned before, this machine is accessible to the public internet. We don’t want this machine processing any incoming requests to our main application, but rather specific views.

To accomplish that I need to introduce you to our fly-replay response header.

`fly-replay` 🔁

Fly Proxy sits between our application and the public internet and is responsible for routing requests to the appropriate Machines based on their location and availability - among many other interesting features!

One of its capabilities includes the ability to redeliver (replay) the original request somewhere else:

in another region;

fly-replay: region=gru

in another app in the same organization;

fly-replay: app=another-app

in a specific machine;

fly-replay: instance=9020ee72fd7948

elsewhere.

    fly-replay: elsewhere=true

  

You can find more details in Playing Traffic Cop with Fly-Replay.

For our Django app, it means we can reroute the request to the serverless machine once we receive a request to a specific Django view.

Aha! Exactly what we’ve been looking for! 😎

Enough of concepts for now… Let’s check how we can implement that in Django.

Django Views as Serverless Functions ✨

For our demo, we simulate a store with purchases and products. A purchase has a list of product items:

We want to offload the receipt’s PDF generation to the serverless function.

These are the main steps we want to achieve:

Client calls our endpoint GET /purchases/serverless-receipts
Fly Proxy receives the request and sends to our Django App
Our Django app receives the request and it goes through the Middleware layer
Our custom Middleware FlyReplayMiddleware checks if the request should be handled in the current Machine or should be replayed in another Machine by evaluating the path requested
GET /purchases/serverless-receipts is our serverless view. Fetch the serverless machine info and add fly-replay: instance=<id> to the response header, where <id> is the Machine id where we should run our serverless view
Fly Proxy receives a response and recognizes the fly-replay header, so it redirects the original request to another Machine, in this case, our serverless machine
The serverless machine receives the request, processes it, and returns the response back to the Fly Proxy
Fly Proxy sends the response back to the client
Win! 🎖

This is the happy path that doesn’t take into consideration other possibilities:

If the serverless machine doesn’t exist: we create a new one and get the info to be added to the fly-replay response header.
If the serverless machine receives a non-serverless view request: the FlyReplayMiddleware will be responsible to reroute the request back to our main application.

We will discuss those scenarios throughout this article and how to handle them.

Time to start coding! 👩🏽‍💻✨

views.py

Let’s start by defining our views:

    # <app>/views.py
# store/views.py
from django.http import HttpResponse
from django.shortcuts import get_object_or_404

from store.helpers import generate_pdf  # helper that generates the PDF

@require_GET
def machines_generate_receipts(request):  
    purchases = Purchase.objects.all()

    buffer = generate_pdf(purchases)  # generate the PDF
    return HttpResponse(buffer, content_type="application/pdf")

@require_GET
def machines_generate_receipt(request, purchase_id):
    purchase = get_object_or_404(Purchase, id=purchase_id)

    buffer = generate_pdf([purchase])  # generate the PDF
    return HttpResponse(buffer, content_type="application/pdf")

  

machines_generate_receipts() generates and renders a PDF file for all the purchases we currently have. machines_generate_receipt() generates and render a PDF file for a specific purchase.

urls.py

The urls are defined as follows:

    # <app>/urls.py
# store/urls.py
from django.urls import path
from .views import machines_generate_receipts

app_name = "store"

urlpatterns = [
    path(
        "purchases/serverless-receipts/",
        machines_generate_receipts,
        name="machines_generate_receipts",
    ),
    path(
        "purchases/serverless-receipts/<int:purchase_id>/",
        machines_generate_receipt,
        name="machines_generate_receipt",
    ),
]

  

We aim to run machines_generate_receipts and machines_generate_receipt views as serverless functions.

settings.py

Now, let’s define in the settings.py the necessary environment variables to create new machines:

    # <project>/settings.py
# fly/settings.py
import environ

env = environ.Env()

FLY_API_HOSTNAME = "https://api.machines.dev"
FLY_API_TOKEN = env("FLY_API_TOKEN")

FLY_IMAGE_REF = env("FLY_IMAGE_REF")
FLY_APP_NAME = env("FLY_APP_NAME")

  

FLY_API_HOSTNAME: the recommended way to connect with Machines API is to use the public API https://api.machines.dev
FLY_API_TOKEN: it’s possible to create a new auth token in your Fly.io dashboard. For local development, you can simply access the token used by flyctl by using:

    fly secrets set FLY_API_TOKEN=$(fly auth token)

  

FLY_IMAGE_REF and FLY_APP_NAME: those are environment variables available to us when our app is deployed on Fly.io.

We also specify the list of views to run as serverless functions:

    # <project>/settings.py
# fly/settings.py
FLY_MACHINES_SERVERLESS_FUNCTIONS = [
    # ("<app>:<path_name>", (<methods>))
    ("store:machines_generate_receipts", ("GET",)),
    ("store:machines_generate_receipt", ("GET",)),
]

  

Now, let’s work with Fly Machines API.

Fly Machines

To keep all the logic around Machines in one place, let’s define a machines.py:

    # <app>/machines.py
# store/machines.py
import requests
from django.conf import settings
from django.urls import resolve, Resolver404

FLY_API_HOSTNAME = settings.FLY_API_HOSTNAME
FLY_API_TOKEN = settings.FLY_API_TOKEN
FLY_APP_NAME = settings.FLY_APP_NAME
FLY_IMAGE_REF = settings.FLY_IMAGE_REF

headers = {
    "Authorization": f"Bearer {FLY_API_TOKEN}",
    "Content-Type": "application/json",
}


def machines_url():
    return f"{FLY_API_HOSTNAME}/v1/apps/{FLY_APP_NAME}/machines"


def create_serverless_machine():
    config = {
        "config": {
            "image": FLY_IMAGE_REF,
            "env": {"IS_FLY_MACHINE_SERVERLESS": "true"},
            "services": [
                {
                    "protocol": "tcp",
                    "internal_port": 8000,
                    "autostop": True,
                    "ports": [
                        {
                            "handlers": ["tls", "http"],
                            "port": 443,
                        },
                        {
                            "handlers": ["http"],
                            "port": 80
                        },
                    ],
                }
            ],
        }
    }
    response = requests.post(
        machines_url(), headers=headers, json=config
    )
    if response.ok:
        return response.json()
    else:
        response.raise_for_status()


def get_or_create_serverless_machine():
    response = requests.get(
        machines_url(),
        headers=headers,
        params={"include_deleted": False},
    )
    if response.ok:
        machines = response.json()
        for machine in machines:
            if (
                machine["config"]["env"].get(
                    "IS_FLY_MACHINE_SERVERLESS", "false"
                )
                == "true"
            ):
                return machine
    else:
        response.raise_for_status()
    return create_serverless_machine()


def is_serverless_function_request(request):
    for pattern, methods in getattr(
        settings, "FLY_MACHINES_SERVERLESS_FUNCTIONS", []
    ):
        request_method = request.method
        try:
            resolved_match = resolve(request.path)
            url_name = resolved_match.url_name
            app_name = resolved_match.app_name
            if (
                f"{app_name}:{url_name}" == pattern
                and request_method in methods
            ):
                return True
        except Resolver404:
            continue
    return False

  

That was a lot of code. Shall we break this down?

Let’s start from the top:

    def create_serverless_machine():
    config = {
        "config": {
            "image": FLY_IMAGE_REF,
            "env": {"IS_FLY_MACHINE_SERVERLESS": "true"},
            "services": [
                {
                    "protocol": "tcp",
                    "internal_port": 8000,
                    "autostop": True,
                    "ports": [
                        {
                            "handlers": ["tls", "http"],
                            "port": 443,
                        },
                        {
                            "handlers": ["http"], 
                            "port": 80
                        },
                    ],
                }
            ],
        }
    }
    response = requests.post(
        machines_url(), headers=headers, json=config
    )
    if response.ok:
        return response.json()
    else:
        response.raise_for_status()

  

create_serverless_machine() creates a new machine using the current Docker image of our Django app (FLY_IMAGE_REF). It spins up a new machine identical to our main app. The configurations worth highlighting are:

    "config": {
    "env": {
        "IS_FLY_MACHINE_SERVERLESS": "true"
    }
    ...
}

  

in config.env: the key/value pair to be set as an environment variable.

We’ll use IS_FLY_MACHINE_SERVERLESS environment variable to filter the machine in which the serverless view should run on. Note: any key/value pair should be set as <string>:<string>, e.g. "IS_FLY_MACHINE_SERVERLESS": "true".

    "config": {
    ...
    "services": [{
            ...
            "autostop": True,
    }]
}

  

in config.services: setting autostop to true

autostop is false by default. When true, the Fly Proxy automatically stops the machine when idle, so you don’t pay for the machine when there is no traffic. That’s the catch!

    def get_or_create_serverless_machine():
    response = requests.get(
        machines_url(),
        headers=headers,
        params={"include_deleted": False},
    )
    if response.ok:
        machines = response.json()
        for machine in machines:
            if (
                machine["config"]["env"].get(
                    "IS_FLY_MACHINE_SERVERLESS", "false"
                )
                == "true"
            ):
                return machine
    else:
        response.raise_for_status()
    return create_serverless_machine()

  

get_or_create_serverless_machine(), as the name suggests, gets the serverless machine or creates a new one. This is done by fetching all non-deleted machines and checking the IS_FLY_MACHINE_SERVERLESS environment variable we defined in create_serverless_machine().

    def is_serverless_function_request(request):
    for pattern, methods in getattr(
        settings, "FLY_MACHINES_SERVERLESS_FUNCTIONS", []
    ):
        request_method = request.method
        try:
            resolved_match = resolve(request.path)
            app_name = resolved_match.app_name
            url_name = resolved_match.url_name
            if (
                f"{app_name}:{url_name}" == pattern
                and request_method in methods
            ):
                return True
        except Resolver404:
            continue
    return False

  

Lastly, is_serverless_function_request() checks FLY_MACHINES_SERVERLESS_FUNCTIONS list from our settings.py and returns weather the current request.path should be handled by our serverless machine.

Now, when a request to a serverless view comes in to our main app, what do we do? And when a request to a non-serverless view comes in to our serverless machine, what should we do?

There’s still a piece of the puzzle missing. 🧩

FlyReplayMiddleware

To wrap up, let’s build the last piece: our custom Middleware.

The Django Middleware operates globally. It acts as an intermediary between the request from the user and the response from the server.

We need a way to intercept the request when it comes in to decide if we should reroute it to another machine.

First, let’s create a middleware.py to define our custom middleware:

    # <app>/middleware.py
# store/middleware.py
import os
import logging

import requests
from django.conf import settings
from django.http import HttpResponse

from .machines import (
    get_or_create_serverless_machine,
    is_serverless_function_request,
)

FLY_APP_NAME = settings.FLY_APP_NAME

class FlyReplayMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if is_serverless_function_request(request):
            if not os.environ.get("IS_FLY_MACHINE_SERVERLESS"):
                try:
                    response = HttpResponse()
                    serverless_machine = (
                        get_or_create_serverless_machine()
                    )
                    response[
                        "fly-replay"
                    ] = f"instance={serverless_machine['id']}"
                    return response
                except requests.exceptions.HTTPError as err:
                    return HttpResponse(
                        "Something went wrong", status=404
                    )
        elif (
            os.environ.get("IS_FLY_MACHINE_SERVERLESS") == "true"
        ):
            response = HttpResponse()
            response["fly-replay"] = f"elsewhere=true"
            return response
        response = self.get_response(request)
        return response

  

and add it to settings.py:

    # <project>/settings.py
# fly/settings.py
MIDDLEWARE = [
    "django.middleware.security.SecurityMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.middleware.common.CommonMiddleware",
    "django.middleware.csrf.CsrfViewMiddleware",
    "django.contrib.auth.middleware.AuthenticationMiddleware",
    "django.contrib.messages.middleware.MessageMiddleware",
    "django.middleware.clickjacking.XFrameOptionsMiddleware",
    "store.middleware.FlyReplayMiddleware",  # ← Added!
]

  

As a reminder, Django applies the middleware in the order it’s defined in the list, top-down.

Now, let’s understand what our custom middleware is doing.

If our main app receives a request to a serverless view, it gets the serverless machine id and adds the fly-replay: instance=<id> to the response header to be replayed on that specific machine. The serverless machine is found based on the existence of IS_FLY_MACHINE_SERVERLESS environment variable.

If our Serverless Machine receives a request to a non-serverless view, it adds the fly-replay: elsewhere=true to the response header to be replayed anywhere else other than the serverless machine.

Otherwise, we just call the view and return the response as per usual.

Here is the table to better visualize the scenarios:

Views	Main App	Serverless Machine
Serverless	add `fly-replay: instance=<id>` to response header	call the view and return a response
Non-Serverless	call view and return a response	add `fly-replay: elsewhere=true` to the response header

That’s it! You made it to the end! 🎉

Try it out 🚀

Last but not least, let’s try this demo out here:

Note: Receipts are generated using reportlab (reportlab==4.0.7).

Final Thoughts

Machines API offers a way to create services that can seamlessly scale down to zero during periods of inactivity.

However, it’s crucial to emphasize that while working directly with the Machines API provides us with greater control over our Machines, it also involves assuming some of the orchestration responsibility.

This post covers creating a serverless Machine on the spot and using it to invoke specific Django views. Note that topics such as deployment, latency and replacing the serverless machine with new versions are not covered in the discussion. Let’s save it for a future blog post, shall we?

Our demo also demonstrates some important and interesting concepts about Fly Machines, in particular, processes, services, Fly Proxy and fly-replay.

Such an approach can also be applied to other use cases previously discussed in this blog. If Django Async Views fall short in preventing the blocking of your main application, and a task queue such as Celery seems like overkill for your not-so-often but resource-intensive or I/O-bound operations, you might want to explore Fly Machines as an alternative.

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to deploy those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

Staging environments on the Fly with GitHub actions

2024-01-04T00:00:00+00:00

Mariusz Felisiak, a Django and Python contributor and a Django Fellow, explores how to create staging environments on the Fly.io with GitHub actions. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

Creating staging environments for testing changes to our apps can be a challenge. This article shows how to use GitHub actions to smoothly create a separate staging environment for each pull request using the fly-pr-review-apps action, which will create and deploy our Django project with changes from the specific pull request. It will also destroy it when it’s no longer needed, such as after closing or merging a pull request. The entire staging process enclosed in one GitHub action, so we don’t have to worry about anything else (NoOps).

Let’s check how it works and why it’s worth using.

Set up

We assume you’ve already set up your Fly.io account, so go ahead and sign in. If you haven’t done that yet, you can sign up to Fly.io.

We also need an existing or new Django project with a fly.toml configuration file. Here are some great resources for getting started with Django or deploying your Django app to Fly.io.

With a project ready, let’s get started!

Basic flow

GitHub action workflows are defined by YAML files in the .github/workflows/ directory of our repository. Let’s add a new flow that will create and deploy staging environment for each pull request. But first we need to create a new repository secret called FLY_API_TOKEN to use for authentication. Go to a GitHub repository page and open:

Settings (tab) → Security → Secrets and variables → Actions → Repository secrets → New repository secret

next, create a new secret called FLY_API_TOKEN with a value from:

fly auth token

It’s also possible to create a new token on the dashboard.

Now, we’re ready to add a new flow. This is how we can define it in the .github/workflows/fly_pr_preview.yml file:

    # .github/workflows/fly_pr_preview.yml

name: Start preview app

on:
  pull_request:
    types: [labeled, synchronize, opened, reopened, closed]

concurrency:
  group: ${{ github.workflow }}-pr-${{ github.event.number }}
  cancel-in-progress: true

permissions:
  contents: read

env:
  FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

jobs:
  preview-app:
    if: contains(github.event.pull_request.labels.*.name, 'PR preview app')
    runs-on: ubuntu-latest
    name: Preview app
    environment:
      name: pr-${{ github.event.number }}
      url: ${{ steps.deploy.outputs.url }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Deploy preview app
        uses: superfly/fly-pr-review-apps@1.2.0
        id: deploy
        with:
          region: waw
          org: personal

  

It’s time to split our configuration up into its component parts:

on → pull_request → types: specifies events on which a new staging project will be deployed:

    pull_request:
  types: [labeled, synchronize, opened, reopened, closed]

concurrency: prevents concurrent deploys for the same PR (Pull Request). The group name contains a PR number and workflow name to create a separate group for each workflow and PR:

    concurrency:
  group: ${{ github.workflow }}-pr-${{ github.event.number }}
  cancel-in-progress: true

  

env: makes the FLY_API_TOKEN secret available to use for authentication:

    env:
  FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

jobs → preview-app → if: skips deploy on PRs without the PR preview app label. Both for safety reasons and to avoid creating a staging environments when no needed:

    if: contains(github.event.pull_request.labels.*.name, 'PR preview app')

  

jobs → preview-app → environment: describes the deployment target to show up in a pull request UI. steps.deploy.outputs.url is filled by the superfly/fly-pr-review-apps and will contain the URL of a deployed staging project:

    environment:
  name: pr-${{ github.event.number }}
  url: ${{ steps.deploy.outputs.url }}

  

jobs → preview-app → steps → with: specifies all inputs that we want to pass to our flow. You can check available options in the README. We pass the Fly.io region and organization as a starting point:

    with:
  region: waw
  org: personal

  

The action configured in this way will deploy an app with the name created according to the following pattern:

pr-{{ PR number }}-{{ repository owner }}-{{ repository name }}

to the: https://{{ app name }}.fly.dev, e.g.

https://pr-9-felixxm-fly-pr-preview-example.fly.dev/

(for PR number 9 in my personal repository called pr-preview-example).

So far so good. However, deploying from scratch a fully functional Django project may require a few more steps such as running migrations or collecting static files. Furthermore, deploying a staging environment in particular may involve even more additional tasks to perform, such as loading fixtures with test data or using a dedicated staging database. The question is how to handle them and where to place each one. Luckily for us, all of them can be handled seamlessly with the fly-pr-review-apps action. The next two sections show you how to do this.

Use Postgres cluster

Using a dedicated staging database is a good practice for test environments. This gives us more control and appropriate separation from the production environment which eliminates a potential data leak vector.

If you don’t have a Postgres cluster specifically for testing purposes you can create one with fly postgres create:

fly postgres create --name pg-fly-pr-staging-preview

Once created, we can specify it in our action (jobs → preview-app → steps → with) using the postgres input:

    # .github/workflows/fly_pr_preview.yml

name: Start preview app

on:
  pull_request:
    types: [labeled, synchronize, opened, reopened, closed]

concurrency:
  group: ${{ github.workflow }}-pr-${{ github.event.number }}
  cancel-in-progress: true

permissions:
  contents: read

env:
  FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

jobs:
  preview-app:
    if: contains(github.event.pull_request.labels.*.name, 'PR preview app')
    runs-on: ubuntu-latest
    name: Preview app
    environment:
      name: pr-${{ github.event.number }}
      url: ${{ steps.deploy.outputs.url }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Deploy preview app
        uses: superfly/fly-pr-review-apps@1.2.0
        id: deploy
        with:
          postgres: pg-fly-pr-staging-preview  # ← Added
          region: waw
          org: personal

  

With that in place, our staging Postgres cluster will be automatically attached to the test app, which will make a DATABASE_URL environment variable available in the test VM.

The next section shows how to perform additional release steps when deploying a staging app.

Additional release steps

Let’s assume that we want to perform additional release steps when deploying our staging environment. For this, we can use a custom Fly.io TOML configuration file dedicated for staging with a release script to run before a deployment. First, make a copy of an existing configuration:

mkdir staging

cp fly.toml staging/fly_staging.toml

Next, create a script (staging/post_deploy.sh) to prepare our staging database:

    # staging/post_deploy.sh
#!/usr/bin/env bash

# Migrate database.
python /code/manage.py migrate
# Load fixtures with test data.
python /code/manage.py loaddata /code/staging/test_groups.json

  

Finally, we need to add release_command to the TOML configuration calling our script:

    # staging/fly_staging.toml

console_command = "/code/manage.py shell"

[build]

[env]
 PORT = "8000"

[deploy]
 release_command = "sh ./staging/post_deploy.sh" # ← Added.

...

and specify the custom TOML configuration in our action (jobs → preview-app → steps → with) using the config input:

    # .github/workflows/fly_pr_preview.yml

name: Start preview app

on:
  pull_request:
    types: [labeled, synchronize, opened, reopened, closed]

concurrency:
  group: ${{ github.workflow }}-pr-${{ github.event.number }}
  cancel-in-progress: true

permissions:
  contents: read

env:
  FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

jobs:
  preview-app:
    if: contains(github.event.pull_request.labels.*.name, 'PR preview app')
    runs-on: ubuntu-latest
    name: Preview app
    environment:
      name: pr-${{ github.event.number }}
      url: ${{ steps.deploy.outputs.url }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Deploy preview app
        uses: superfly/fly-pr-review-apps@1.2.0
        id: deploy
        with:
          config: staging/fly_staging.toml  # ← Added
          postgres: pg-fly-pr-staging-preview
          region: waw
          org: personal

  

With this small effort we have staging database created on Fly! 🚀

🚨 Be aware that release_command is run on a temporary VM and cannot modify the local storage or state. It’s fine to run database operations but not to perform release steps that attempt to modify a local storage, e.g. collecting static files. Such steps should be added to the Dockerfile. For example, if we want to collect static files, then add the collectstatic management command to the Dockerfile:

    ARG PYTHON_VERSION=3.10-slim-bullseye

FROM python:${PYTHON_VERSION}

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

RUN mkdir -p /code

WORKDIR /code

COPY requirements.txt /tmp/requirements.txt
RUN set -ex && \
    pip install --upgrade pip && \
    pip install -r /tmp/requirements.txt && \
    rm -rf /root/.cache/
COPY . /code

# ↓ Added ↓
RUN set -ex && \
    python /code/manage.py collectstatic --noinput

EXPOSE 8000

CMD ["gunicorn", "--bind", ":8000", "--workers", "2", "hello_pr_preview_example.wsgi"]

  

Let’s take a look how it works:

We did it 🚀 Check other options and give it a spin!

Serverless Functions in Django With Fly Machines

2023-11-23T00:00:00+00:00

In this post we demonstrate how to use Fly Machines to run Serverless Functions using Django custom commands. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

In the Serverless model, despite the name, there are still servers under the hood.

Serverless is a broader concept that encompasses various cloud computing services where we don’t manage the infrastructure. Instead, it’s abstracted away from us - and we like that! The cloud provider executes the code and shuts down the resources when they are no longer needed - aka “scale to zero”. That lets us focus on the application code instead of managing the infrastructure. That’s the case if you are running your Django app on Fly.io, more specifically on Machines Platform and your app is configured to stop and start automatically.

In this post, we will use the power of Fly Machines to run a long-running task as a serverless function using Django custom commands.

Let’s dive right in.

Serverless Functions 🚀

Serverless functions - aka Function-as-a-Service (FaaS) - are a key component of Serverless architecture. The cloud provider dynamically manages the allocation and execution of server resources.

But here is the catch.

Those edge function services from your favorite cloud provider usually require managing functions in a separate manner. Meaning, a lot of additional configuration - yes, I’m talking to you “permissions and policies”. On top of that, functions are often tied to specific cloud providers, which can make porting to other platforms difficult.

Like any other technology, serverless functions have other disadvantages like:

Cold Starts: the first request after idle time usually has a higher latency due to the time it takes to initialize a function instance.
Statelessness: maintaining application state can be challenging and often requires external services.
Limited Execution Time: most platforms impose a maximum execution time for each function invocation.

But they also have their advantages:

Cost Efficiency: we only pay for the time our functions are running, not for the idle server time.
No Server Management: there is no need to provision or maintain servers.
Scalability: automatically scales with the application’s needs.
Quick Deployments: we can deploy and update functions more rapidly compared to traditional applications.

Those are valid points for most of the cloud providers.

The question is: how is the Django community using Serverless Functions nowadays?

Serverless Functions in Django

When we talk about integrating Serverless Functions into our Django app, we are typically discussing about one of the two scenarios:

Replacing Traditional Server Infrastructure: the entire Django application, or significant parts of it, are hosted on a serverless platform. This approach can be complex due to the stateless nature of serverless functions and the typically stateful nature of web applications like those built with Django.
Hybrid Approach with Serverless Components: Certain components or aspects of the Django application are offloaded to serverless functions. Common use cases include handling file uploads, sending emails, or performing other resource-intensive tasks. This approach allows for scaling those parts of the application that require it without needing to scale the entire infrastructure.

One popular package used in the 1st scenario is Zappa. Zappa simplifies the deployment of Python web applications, like those built in Django, on AWS Lambda and API Gateway. In a nutshell, Zappa packages our Django app, along with its dependencies, into a format compatible with AWS Lambda. Zappa then deploys this package, setting up API Gateway to handle web requests and route them to AWS Lambda.

Yet, in this post, we focus on the 2nd scenario and demonstrate how we can leverage Fly Machines to offload the work to serverless functions using Django commands.

But first, do you know about Fly Machines?

Fly Machines 🤖

Fly Machines - sometimes mentioned as just Machines around here - are lightweight VMs based on Firecracker that can boot instances in about 300ms. Machines are the building blocks for the V2 Fly Apps Platform.

Machines deployed by Fly Launch are managed as a group and run from the same image and configuration.

In a previous article, we deployed our Django app and Celery worker processes on Fly Machines running multiple process groups in an App:

This implementation works as expected. Our web app stops when there is no traffic, as a result of the following configuration:

    # fly.toml

[http_service]
  ...
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ["app"]

  

And our Celery worker is running in its own process with dedicated resources and possibility of scaling independently. Essentially, our worker stays active even when no tasks are being executed.

But what if we want to start/stop other processes the same way we do for our web server? You might ask.

Well, unfortunately this is not a trivial task since this feature relies on an HTTP service - it automatically starts and stops existing Machines based on incoming requests. This is also true if we want to spin up any other non-HTTP process alongside our app.

So, how can we leverage the “scale to zero” feature of Fly.io to spin up processes to be used in our Django app?

Fly Machines API to the Rescue

We got you covered!

Fly Machines provides a REST API to interact with our Machines. That’s exactly what we need! With that, we have more control over our Machines.

The easiest and recommended way to connect to the Machines API is to use public API https://api.machines.dev/.

It’s also possible to use the private Machines API over a Wireguard VPN. This requires more setup.

For now, we will stick to the public Machines API.

Django Commands as Serverless Functions ✨

For the purpose of this post, we are building a simple app that analyzes the sentiment of a given English text, returning the polarity and subjectivity values and based on that if the text has a positive, negative or neutral sentiment:

These are the steps we want to achieve:

Client calls our API POST /analyze-sentiment;
Our view calls FlyMachine.run(<cmd>);
FlyMachine.run(<cmd>) calls Machines API POST /apps/<app_name>/machines, spins up a new Machine (e.g. Machine #1) by running the specified Django command, for our example, python manage.py sentiment <text_id> <fly_machine_id>;
Machine #1 start its own process, executes the function (sentiment command) and Machine is then destroyed when the process is complete;
Client calls our API GET /get-analysis/<machine_id> to get the analysis info;
Our view calls FlyMachine.get_or_update_state();
FlyMachine.get_or_update_state() calls Machines API GET /apps/<app_name>/machines/<machine_id> (if Machine is not destroyed yet) to get the state and update our machine object.

Note that when creating a new machine, the only required parameter is the Docker image to run. When our app is deployed on Fly.io, information about the runtime environment is available to us. One of them is the FLY_IMAGE_REF which is the Docker image running in the current container. This is particularly interesting for us since we want to use the same image to create a Machine to scale up-or-down.

Let’s check how we can implement it.

Fly Machine via Machines API

We assume you’ve already set up your Fly.io account, so go ahead and sign in. If you haven’t done that yet, you can sign up to Fly.io.

We need an access token (FLY_API_TOKEN) in order to access the Machines API. For local development, your can access the token used by flyctl using:

fly auth token

It’s also possible to create a new auth token in your Fly.io dashboard.

Before our app is deployed to Fly.io, we need to make the token available as a secret:

fly secrets set FLY_API_TOKEN=$(fly auth token)

fly secrets set FLY_API_TOKEN=<your_newly_created_access_token>

Machines App

Our project relies on a Django machines app.

The machines app has all the logic around Fly Machines and where all the Machines’ information lives.

We start by defining the FlyMachine model:

    # machines/models.py

class FlyMachine(models.Model):
    class MachineState(models.IntegerChoices):
        UNCREATED = 0, _("uncreated")
        CREATED = 1, _("created")
        STARTING = 2, _("starting")
        STARTED = 3, _("started")
        STOPPING = 4, _("stopping")
        STOPPED = 5, _("stopped")
        REPLACING = 6, _("replacing")
        DESTROYING = 7, _("destroying")
        DESTROYED = 8, _("destroyed")

    machine_id = models.CharField(
        max_length=20, null=True, blank=True
    )
    instance_id = models.CharField(
        max_length=30, null=True, blank=True
    )
    app_name = models.CharField(max_length=100)
    image_ref = models.CharField(max_length=80)
    region_id = models.CharField(
        max_length=3, null=True, blank=True
    )
    state = models.IntegerField(
        choices=MachineState.choices,
        default=MachineState.UNCREATED,
    )
    created_at = models.DateTimeField(auto_now_add=True)

    class Meta:
        ordering = ['-created_at']

    def __str__(self):
        return f"Machine #{self.id}"

  

The app_name and image_ref are the only required fields. app_name is the unique application name on Fly.io (e.g. fly-machines-django) and it’s necessary to define in which Fly App the Machine will be created at. The image_ref is the Docker image used to boot our machine.

We implement run() and get_or_update_state() methods in FlyMachine:

    # machines/models.py

class FlyMachine(models.Model):
    ...

    def run(self, cmd):
        try:
            data = FlyMachinesAPIService.create_machine(
                self.app_name, self.image_ref, cmd
            )
        except requests.exceptions.HTTPError as err:
            logger.error("HTTPError: %s", err)
            raise
        self.result = data
        self.machine_id = data.get("id")
        self.instance_id = data.get("instance_id")
        self.region_id = data.get("region")
        machine_state = data.get("events")[0][
            "status"
        ]  # last event's status
        self.state = next(
            state
            for state in FlyMachine.MachineState
            if state.label == machine_state
        )
        self.save()

    def get_or_update_state(self):
        if self.state != FlyMachine.MachineState.DESTROYED:
            try:
                data = FlyMachinesAPIService.get_machine(
                    self.app_name, self.machine_id
                )
            except requests.exceptions.HTTPError as err:
                logger.error("HTTP error occurred: %s", err)
                raise
            self.result = data
            machine_state = data.get("events")[0][
                "status"
            ]  # last event's status
            self.state = next(
                state
                for state in FlyMachine.MachineState
                if state.label == machine_state
            )
            self.save()
        return self.state

  

The run() method calls Machines API to create a new Machine and save the appropriate results into our FlyMachine instance.

The get_or_update_state() method will be responsible to fetch and update the latest state of the Machine.

For our example, the FlyMachinesAPIService class handles the HTTP request logic:

    # machines/services.py

FLY_API_HOSTNAME = os.environ.get("FLY_API_HOSTNAME", "https://api.machines.dev")
FLY_API_TOKEN = os.environ.get("FLY_API_TOKEN")

headers = {
    "Authorization": f"Bearer {FLY_API_TOKEN}",
    "Content-Type": "application/json",
}


class FlyMachinesAPIService:
    @staticmethod
    def create_machine(app_name, image_ref, cmd):
        config = {
            "config": {
                "image": image_ref,
                "auto_destroy": True,
                "processes": [{"cmd": cmd.split(" ")}],
            }
        }
        url = f"{FLY_API_HOSTNAME}/v1/apps/{app_name}/machines"
        response = requests.post(
            url, headers=headers, json=config
        )
        if response.ok:
            return response.json()
        else:
            response.raise_for_status()

    @staticmethod
    def get_machine(app_name, machine_id):
        url = f"{FLY_API_HOSTNAME}/v1/apps/{app_name}/machines/{machine_id}"
        response = requests.get(url, headers=headers)
        if response.ok:
            return response.json()
        else:
            response.raise_for_status()

  

All requests to Machines API must include the Fly API Token in the HTTP Header by adding the Authorization to the headers:

    Authorization: Bearer <FLY_API_TOKEN>

  

When a FlyMachine instance is created:

    machine = FlyMachine.objects.create(
    app_name=FLY_APP_NAME,
    image_ref=FLY_IMAGE_REF,
)

  

a unique id is assigned to it. At this point we have all the necessary information to create a new Machine: app_name, image_ref and cmd. cmd is then passed as a parameter to the run method:

    machine.run(
    cmd=f"python manage.py sentiment {text.id} {machine.id}"
)

  

In create_machine(), we create a new Machine by calling POST /apps/{app_name}/machines. A config object defines the Machine’s configuration:

    config = {
    "config": {
        "image": image_ref,
        "auto_destroy": True,
        "processes": [{"cmd": cmd.split(" ")}],
    }
}

  

image: Docker image to run (required)
auto_destroy: if True, the Machine will be destroyed once its process is complete. Default is False.
cmd: an array of strings. These will be the arguments passed to the entrypoint to be run.

These properties are enough to run our Django command in a new Machine. You can learn about other properties here.

Besides that, in get_machine(), we fetch the information about a Machine by calling GET /apps/{app_name}/machines/{machine_id}. Once the Machine’s state is DESTROYED, it no longer exists. If state is anything but DESTROYED, we fetch the Machine info and update its state.

Sentiment App

Now, let’s create a sentiment app:

    # sentiment/models.py

class TextAnalysisForm(forms.ModelForm):
    class Meta:
        model = TextAnalysis
        fields = ['text'] 


class TextAnalysis(models.Model):
    text = models.TextField()
    polarity = models.FloatField(null=True, blank=True)
    subjectivity = models.FloatField(null=True, blank=True)
    machine = models.OneToOneField(
        FlyMachine,
        on_delete=models.SET_NULL,
        related_name="text_analysis",
        null=True,
        blank=True,
    )

    class Meta:
        ordering = ["-id"]

    def __str__(self):
        return f"Text #{self.id}"

  

Our model have a OneToOneField to our FlyMachine. For our demo, each analysis (“task”) runs on one Machine but you can implement it in a way that multiple analysis (“tasks”) are executed in a single Machine.

Our view might look like this:

    # sentiment/views.py

FLY_APP_NAME = os.environ.get("FLY_APP_NAME")
FLY_IMAGE_REF = os.environ.get("FLY_IMAGE_REF")


@require_http_methods(["GET", "POST"])
def analyze_sentiment(request):
    if request.method == "POST":
        form = TextAnalysisForm(request.POST)
        if form.is_valid():
            try:
                with transaction.atomic():
                    text_analysis = form.save()
                    machine = FlyMachine.objects.create(
                        app_name=FLY_APP_NAME,
                        image_ref=FLY_IMAGE_REF,
                    )
                    machine.run(
                        cmd=f"python manage.py sentiment {text_analysis.id} {machine.id}"
                    )
                    text_analysis.machine = machine
                    text_analysis.save()
                    template_name = "partials/analysis.html"
                    context = {"texts": [text_analysis]}
            except requests.exceptions.HTTPError:
                template_name = "partials/machine_error.html"
                context = {
                    "error_message": "An error occurred: Machine was not created."
                }
    else:
        text_analysis = TextAnalysis.objects.all()[:10]
        form = TextAnalysisForm()
        template_name = "analyze_sentiment.html"
        context = {"form": form, "texts": text_analysis}

    return render(request, template_name, context)


@require_GET
def get_analysis(request, machine_id):
    machine = FlyMachine.objects.get(machine_id=machine_id)
    machine_destroyed = FlyMachine.MachineState.DESTROYED
    if machine.state != machine_destroyed:
        try:
            if machine.get_or_update_state() != machine_destroyed:
                return render(
                    request,
                    "partials/analysis_running.html",
                    {"text": machine.text_analysis},
                )
        except requests.exceptions.HTTPError:
            return render(
                request,
                "partials/machine_error.html",
                {
                    "error_message": "An error occurred: Machine was not updated."
                },
            )
    return render(
        request,
        "partials/analysis_done.html",
        {"text": machine.text_analysis},
    )

  

Finally, let’s implement our sentiment custom command that serves as our Serverless function. Add a new sentiment.py file to the sentiment/management/commands/ folder:

    # sentiment/management/commands/sentiment.py

from django.core.management.base import BaseCommand

from sentiment.helpers import sentiment_analysis


class Command(BaseCommand):
    help = "Analyze sentiment from a TextAnalysis' blob of text."

    def add_arguments(self, parser):
        parser.add_argument(
            "text_id",
            type=int,
            help="TextAnalysis id where text is analyzed.",
        )
        parser.add_argument(
            "fly_machine_id",
            type=int,
            help="FlyMachine id where the task is performed.",
        )

    def handle(self, *args, **kwargs):
        text_id = kwargs["text_id"]
        fly_machine_id = kwargs["fly_machine_id"]

        sentiment_analysis(text_id)  # run the sentiment analysis

        self.stdout.write(
            self.style.SUCCESS(
                f" Task FlyMachine ID {fly_machine_id} is complete!"
            )
        )

  

Tada! 🚀 We’ve spin up a new Machine, run our python manage.py sentiment command and the machine was destroyed right after the command was complete. No extra configuration, just the good and old Django code we all know and love! 💓

The app is available here if you want to try it out!

Note that we’re using django-htmx to fetch and update the status of the machine dynamically. Partial rendering is not in the scope of this post but you can learn more about it on this blog post.

Do you have any interesting use cases? Share with us at the Fly Community.

Final Thoughts: Limitations and Flaws

⚠️ This approach might not be the solution you’ve been waiting for that project, so consider this section as a heads-up!

The primary design of Django revolves around being a monolithic web framework. It enables rapid development by providing a set of ready-to-use tools and conventions. In addition to that, maintenance and updates are more straightforward. This serverless approach keeps your Django project as it is, monolithic, and not a “separate service” and one more thing to manage.

But not everything is sunshine and rainbows. ☀️ 🌈

This approach may pose challenges in terms of scalability and flexibility as our application evolves. The Machine’s boot time depends on the size of our image, thus complex and large images can be a lot slower to start. In this approach, a single “task” might not be the ideal use-case but if you have multiple long-running tasks to be executed once a day, once a week, etc. you might have much better results.

Additionally, you should take into account potential denial of service (DoS) attacks, especially when dealing with anonymous users. A DoS attack can occur when a user overwhelms the application’s resources or exploits vulnerabilities to disrupt its normal functioning. Consider mitigation strategies such as implementing rate limiting, optimizing database queries, using cache mechanisms, always monitoring your application for unusual traffic patterns and keeping your project upgraded to address potential vulnerabilities.

This is still a proof-of-concept and a very minimal example. There are a lot more to cover for more complex cases. Let’s keep the discussions up!

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to deploy those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

New goodies in Django 5.0

2023-11-14T00:00:00+00:00

Mariusz Felisiak, a Django and Python contributor and a Django Fellow, shares with us his personal favorites from a “deluge” of exciting new features in Django 5.0. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

As planned, after 8 months of intensive development, the first alpha and beta versions of Django 5.0 are out! Almost 700 commits were merged to this release. 204 people 💗 and even more unnamed heroes, dedicated their time and efforts to make the first in the 5.X series the best Django ever 🎉 Let’s explore a “deluge” of amazing features added in this version.

Generated fields

As always, ORM features are firmly on the new features map. Let’s start with the generated fields which are truly game-changing. For many years Django developers struggled with the following questions:

Where to keep reusable properties based on model objects?
How to make them available in all Django components?

Django 5.0 brings the answer to these needs, the new GeneratedField! It allows creation of database-generated fields, with values that are always computed by the database itself from other fields. Database expressions and functions can also be used to make any necessary modifications on the fly. Let’s find out how it works in practice.

Suppose we have fields that are often used concatenated together like first_name and last_name. Implementing a model’s QuerySet or Manager was the best option to share annotations, such as full_name, in Django versions < 5.0. We discussed this in the article about organizing database queries. As a short reminder, it’s possible to change the default manager (objects) and extend available fields by annotations:

    from django.db import models
from django.db.models.functions import Concat


class Order(models.Model):
    person = models.ForeignKey("Person", models.CASCADE)
    ...


class PersonQuerySet(models.QuerySet):
    def with_extra_fields(self):
        return self.annotate(
            full_name=Concat(
                "first_name", models.Value(" "), "last_name",
            ),
        )


class Person(models.Model):
    first_name = models.CharField(max_length=255)
    last_name = models.CharField(max_length=255)

    objects = PersonQuerySet.as_manager()

    def __str__(self):
        return f"{self.first_name} {self.last_name}"

  

Adding annotations in custom QuerySets allows sharing common calculations for model objects:

python3 manage.py shell

    >>> from order.models import Order, Person
>>> Person.objects.with_extra_fields().filter(full_name="Joe Doe")
<PersonQuerySet [<Person: Joe Doe>]>
>>> mark = Person.objects.with_extra_fields().get(full_name="Mark Doe")
>>> mark.full_name
'Mark Doe'
# It's not available on relationship :(
>>> Order.objects.filter(person__full_name="Catherine Smith")
    ...
    raise FieldError(
django.core.exceptions.FieldError: Unsupported lookup 'full_name'
for ForeignKey or join on the field not permitted.

  

However, QuerySet annotations have limitations. First, the full_name is only available to objects returned from the .with_extra_fields() method, so we must remember to use it. It’s also not available for the self instance inside the model methods because it’s not a Person attribute. For example, it cannot be used in Person.__str__(). Furthermore, the values are calculated every time, which can be a problem for complex computations.

GeneratedField makes providing values based on other fields really seamless. Its value is automatically set each time the model is changed by using the given database expression. Moreover, the db_persist parameter allows deciding if it should be stored or not:

db_persist=True: means that the values are stored and occupy storage like any normal column. On the other hand, they are only calculated when the values change, so we avoid additional calculations when fetching the data. On some databases (MySQL, Postgres, SQLite) they can even be indexed!
db_persist=False: means the column doesn’t occupy storage, but its values are calculated every time.

Database support for using different expressions and options may vary. For example, Postgres doesn’t support virtual column, so the db_persist must be set to True. SQLite and MySQL support both virtual and stored generated fields.

Coming back to our “Full name” example. In Django 5.0, we can get rid of all disadvantages of previous approach and define a GeneratedField with the expression used in the full_name annotation:

    from django.db import models
from django.db.models.functions import Concat


class Order(models.Model):
    person = models.ForeignKey("Person", models.CASCADE)
    ...

class Person(models.Model):
    first_name = models.CharField(max_length=255)
    last_name = models.CharField(max_length=255)
    # ↓ Our new GeneratedField based on an expression ↓
    full_name = models.GeneratedField(
        expression=Concat(
            "first_name", models.Value(" "), "last_name"
        ),
        output_field=models.CharField(max_length=511),
        db_persist=True,
    )

    def __str__(self):
        # There is no need to re-implement the same logic anymore!
        return self.full_name

  

Now full_name is a proper Person attribute and can be used in all Django components as any other field:

python3 manage.py shell

    >>> from order.models import Order, Person
>>> Person.objects.filter(full_name="Joe Doe")
<PersonQuerySet [<Person: Joe Doe>]>
>>> mark = Person.objects.get(full_name="Mark Doe")
>>> mark.full_name
'Mark Doe'
# It can be used on relationships.
>>> Order.objects.filter(person__full_name="Catherine Smith")
<QuerySet [<Order: Order object (1)>, <Order: Order object (4)>]>

  

Let’s move on to another amazing new ORM feature that was first requested over 18 (yes, eighteen!) years ago.

Database-computed default values

Database-computed default values were the last significant blocker in fully expressing the structure of Django models and relationship in the database. In Django versions < 5.0, Field.default was the only option for setting the default value for a field, however it’s calculated on Python-side and passed as a parameter when adding a new row. As a consequence, it’s not visible from the database structure perspective and it’s not set when adding rows directly in the database. This also creates a risk of data inconsistency if we take database and network latency into account, for example, when we want to default to the current point in time. Moreover, it’s not visible for people who only have direct access to the database like database administrators or data scientists.

Django 5.0 supports database-computed default values via the new Field.db_default option. This is a panacea for all the concerns related to the previous approach as it allows us to define and compute default values in the database. It accepts literal values and database functions. Let’s inspect a practical example:

    from decimal import Decimal

from django.db import models
from django.db.models.functions import Now


class Order(models.Model):
    person = models.ForeignKey("Person", models.CASCADE)
    created = models.DateTimeField(db_default=Now())
    priority = models.IntegerField(db_default=0)
    ...

  

Form field group rendering

The third great feature that I’d like to highlight is the concept of form field group rendering introduced in Django 5.0, which made form field rendering simpler and more reusable.

Form fields are often rendered with many of their related attributes such as help texts, labels, errors, and widgets. Every project has its own HTML structure and preferred way of rendering form fields. Let’s assume that we use <div> based field templates (like Django) in order to help users with assistive technologies (e.g. screen readers). Our preferred way to render a field could look like this:

    <div class="form-field">
  {{ field.label_tag }}
  {% if field.help_text %}
    <p class="help" id="{{ field.auto_id }}_helptext">
      {{ field.help_text|safe }}
    </p>
  {% endif %}
  {{ field.errors }}
  {{ field }}
</div>

  

It’s exactly the same as Django’s default template for the new as_field_group() method, so now we can simplify it to the:

    <div class="form-field">{{ field.as_field_group }}</div>

  

By default, as_field_group() uses the "django/forms/field.html" template that can be customized on a per-project, per-field, or per-request basis. This gives us a lot of flexibility. Check out the Django documentation for all useful attributes that we can use on custom templates.

Closing thoughts

Django 5.0 with tons of new features is a great start to the 5.X series. Besides the three magnificent enhancements described in this article, Django 5.0 contains outgoing improvements in accessibility and asynchronous areas and many more that can be found in release notes.

What are your personal favorites? Try Django 5.0 and share!

Smooth Database Changes in Blue-Green Deployments

2023-09-14T00:00:00+00:00

Mariusz Felisiak, a Django and Python contributor and a Django Fellow, explores how to make smooth blue-green deployments using advanced migration tools. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

Blue-green deployment is a technique of releasing changes to a project by gradually transferring users to a new version. It provides an isolation between the current “blue” environment and the new “green” environment. Blue nodes are sequentially upgraded to the new “green” version, so for the entire time of deployment we have both environments working simultaneously. It’s quite a challenge to keep both versions running smoothly and to keep downtime as short as possible, with getting close to the mythical zero-downtime.

Changes in an app logic are causing changes in Django models and in the database structure. Therefore, it’s really important to use the right strategy to make the new version backward compatible from the database perspective. Luckily for us, the migrations framework makes it possible in Django.

This article will discuss:

How migrations work?
How to use advanced migration tools to make database transitions in the most efficient way?
How does migration fit into the blue-green strategy?

Let’s start with a brief introduction to the migrations framework.

Migrations framework

The migrations framework is a way of propagating changes in Django models to the database. It makes web development with Django more accessible because, as developers, we no longer need to know, write, and maintain SQL statements with data definitions (known as DDL statements - Data Definition Language). Believe me, maintaining SQL statements can be really painful. Luckily for us, we don’t have to do that anymore.

Let’s explore how it works.

Workflow

The standard workflow for changing our data definitions has three steps:

First, make a change in a model definition. For example, we can add a new field called price to the Book model:

    # bookstore/models.py

from django.db import models


class Font(models.Model):
    name = models.CharField(max_length=255)


class Author(models.Model):
    name = models.CharField(max_length=255)


class Book(models.Model):
    title = models.CharField(max_length=1023)
    description = models.TextField()
    pages = models.IntegerField()
    isbn = models.IntegerField()
    authors = models.ManyToManyField("Author")
    font = models.ForeignKey(
        "Font", on_delete=models.CASCADE, null=True
    )
    # ↓ New field ↓
    price = models.DecimalField(
        max_digits=8, decimal_places=2, null=True
    )

  

Second, run the makemigrations command to generate a new migration file:

python3 manage.py makemigrations

Migrations for 'bookstore':
  bookstore/migrations/0002_book_price.py
    - Add field price to book

Migration files describe what kind of changes are needed in our database structure and what kind of changes has been made in model definitions. That’s what it looks for adding a new field:

    # bookstore/migrations/0002_book_price.py

from django.db import migrations, models


class Migration(migrations.Migration):
   dependencies = [
       ("bookstore", "0001_initial"),
   ]

   operations = [
       migrations.AddField(
           model_name="book",
           name="price",
           field=models.DecimalField(
               decimal_places=2, max_digits=8, null=True
           ),
       ),
   ]

  

dependencies - list of migrations it depends on, e.g. the previous migration name.
operations - list of operations to perform, in our case it is AddField(). The Django documentation describes all kind of core operations.
initial - whether the migration is the first initial migration of an app.

The last step is to propagate changes into our database by running the migrate command.

python3 manage.py migrate

Operations to perform:
 Apply all migrations: admin, auth, bookstore, contenttypes, sessions
Running migrations:
 Applying bookstore.0002_book_price... OK

We now have a new column in the Book table 🚀 .

Fortunately, there are many database operations that are backward compatible and don’t require any special treatment during the blue-green deployments. Django handles them really efficiently in the standard migrations flow, it includes

adding a model (table in the database),
adding a nullable field (column in the database),
removing an index, or
removing a constraint.

We are not discussing a database locks here, which may be required for some of the database structure changes.

Unfortunately, some database transformations are backward incompatible and we need to choose the right strategy and tools to keep them smooth from deployment perspective, it includes

removing a field,
removing a model, or
adding a non-nullable field.

Let’s check what advanced tools are built into Django and how we can use them.

Separate database and project state

What we want to do is to separate model definitions from the corresponding database structure during the blue-green deployments. We want to ensure that removed fields (or models) are no longer used on new “green” nodes, while at the same time the required database columns (or tables) are available for old “blue” nodes.

Django’s answer to this problem is a special and extremely powerful migration operation called SeparateDatabaseAndState(). It allows to separate database and project state, so to separate changes that should be made in the database structure from the changes that are recognized by Django as being made in the state of models. It allows us to persuade Django that project operations were made as it expects and in the same time no SQL statements are issued on the database.

    SeparateDatabaseAndState(
    database_operations=[...],
    state_operations=[...],
)

  

It accepts two lists of operations:

database_operations - list of operations to apply to the database,
state_operations - list of operations to apply to the project state.

Let’s find out how we can use it in practice for removing a field and a model.

Full example

We need an existing or new Django project. Here are some great resources for getting started with Django or deploying your Django app to Fly.io.

With a project ready, let’s get started!

Suppose that we have a bookstore site, that provides various information about available books. In our example, we’ll use the same models (Font, Author, and Book) as defined in the Workflow paragraph. What if, at some point, we realize that collecting information about fonts used in books are difficult and unnecessary. It’s not important for our clients, so we would like to simplify our models definition by removing the Font model and the Book.font field.

The following steps will show you how we can perform this potentially backward incompatible transition in a blue-green deployment by using the SeparateDatabaseAndState() operation.

First step is to remove all logic related to the Font from our project.
Secondly, remove the Font model and the Book.font field:

    --- a/bookstore/models.py
+++ b/bookstore/models.py
@@ -1,10 +1,6 @@
 from django.db import models


-class Font(models.Model):
-    name = models.CharField(max_length=255)
-
-
 class Author(models.Model):
     name = models.CharField(max_length=255)

@@ -15,9 +11,6 @@ class Book(models.Model):
     pages = models.IntegerField()
     isbn = models.IntegerField()
     authors = models.ManyToManyField("Author")
-    font = models.ForeignKey(
-        "Font", on_delete=models.CASCADE, null=True
-    )
     price = models.DecimalField(
         max_digits=8, decimal_places=2, null=True
     )


  

Next, run the makemigrations command to generate a new migration file:

python3 manage.py makemigrations

Migrations for 'bookstore':
  bookstore/migrations/0003_remove_book_font_delete_font.py
    - Remove field font from book
    - Delete model Font

makemigrations generates a new migration with two operations: RemoveField() and DeleteModel():

    # bookstore/migrations/0003_remove_book_font_delete_font.py

from django.db import migrations


class Migration(migrations.Migration):
    dependencies = [
        ("bookstore", "0002_book_price"),
    ]

    operations = [
        migrations.RemoveField(
            model_name="book",
            name="font",
        ),
        migrations.DeleteModel(name="Font"),
    ]

  

Both operations are backward incompatible and deploying this migration as it is would break the old “blue” environments, because the underlying table and column would be deleted. Using the sqlmigrate command we can check SQL statements that will be issued (output for PostgreSQL):

python3 manage.py sqlmigrate bookstore 0003_remove_book_font_delete_font

BEGIN;
--
-- Remove field font from book
--
ALTER TABLE "bookstore_book" DROP COLUMN "font_id" CASCADE;
--
-- Delete model Font
--
DROP TABLE "bookstore_font" CASCADE;
COMMIT;

These operations describe changes that we made in models, so we can wrap them with the SeparateDatabaseAndState() operation by moving to the state_operations, at the same time leaving database_operations empty to avoid changes in the database structure, like so:

    # bookstore/migrations/0003_remove_book_font_delete_font.py

from django.db import migrations


class Migration(migrations.Migration):
    dependencies = [
        ("bookstore", "0002_book_price"),
    ]

    operations = [
        migrations.SeparateDatabaseAndState(
            # Operations in the project state.
            state_operations=[
                migrations.RemoveField(
                    model_name="book",
                    name="font",
                ),
                migrations.DeleteModel(name="Font"),
            ],
            # No changes in the database.
            database_operations=[],
        ),
    ]

  

Using the sqlmigrate command again confirms that no SQL statements will be issued:

python3 manage.py sqlmigrate bookstore 0003_remove_book_font_delete_font

BEGIN;
--
-- Custom state/database change combination
--
-- (no-op)
COMMIT;

We can now safely deploy our new “green” version to all instances in a fully backward compatible manner from a database perspective, as nothing has changed in the database structure 🎉 .

Check out the bluegreen deployment strategy on Fly.io! 💙 → 💚

Once the new version is deployed to all instances and when we are sure that the underlying table and column are unused, we can actually remove them from the database.

To do this, we need a new blank migration file that will be structured like other migration files. To generate such a file, use the --empty flag to the makemigrations command. In order to follow good practice, and to avoid migration names based on timestamp, I’d recommend to also pass the --name flag, e.g. "remove_book_font_delete_font_from_db" which makes it clear what this migrations will do:

python3 manage.py makemigrations bookstore \
--empty \
--name remove_book_font_delete_font_from_db

Migrations for 'bookstore':
  bookstore/migrations/0004_remove_book_font_delete_font_from_db.py

The dependencies are already filled and we have an empty list of operations ready to be use:

    # bookstore/migrations/0004_remove_book_font_delete_font_from_db.py

from django.db import migrations

class Migration(migrations.Migration):
    dependencies = [
        ("bookstore", "0003_remove_book_font_delete_font"),
    ]

    operations = []

We want to perform the same operations, but this time in the database, not in the project state. Therefore, we will use SeparateDatabaseAndState() again but with SQL statements that describe RemoveField() and DeleteModel() operations in the database_operations and an empty list of state_operations. The modified migration file 0004_remove_book_font_delete_font_from_db.py should look like this:

    # bookstore/migrations/0004_remove_book_font_delete_font_from_db.py

from django.db import migrations


class Migration(migrations.Migration):
    dependencies = [
        ("bookstore", "0003_remove_book_font_delete_font"),
    ]

    operations = [
        migrations.SeparateDatabaseAndState(
            # No changes in the project.
            state_operations=[],
            # Operations to be performed on the database.
            database_operations=[
                # SQL statement for RemoveField().
                migrations.RunSQL(
                    sql=(
                        'ALTER TABLE "bookstore_book" '
                        'DROP COLUMN "font_id" CASCADE;'
                    ),
                ),
                # SQL statement for DeleteModel().
                migrations.RunSQL(
                    sql='DROP TABLE "bookstore_font" CASCADE;',
                ),
            ],
        ),
    ]

  

We can now safely deploy our latest “green” version to all instances in a fully backward compatible manner as the deleted table and column are no longer used by any node after the previous deployment 💙 → 💚. Voilà ✨

Closing Thoughts

Separating database and project states gives us highly flexible pattern for propagating database changes. It is not only handy for blue-green deployments, but it also allows for describing potentially not feasible transitions in the most efficient way. Frequently without any data migration, without any intermediate steps, atomic, and even reversible. Some use cases are:

changing a ManyToManyField to use a through model,
changing a custom through model to auto-generated intermediate table,
moving models between apps, etc.

Try it and share!

Remember to check out the new and shiny bluegreen deployment strategy on Fly.io!

Celery Async Tasks on Fly Machines

2023-08-22T00:00:00+00:00

In this post we offload the work from our application and run async tasks using Celery workers on Fly Machines. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

It’s time… We are finally talking Celery! 🎉

We’ve been discussing about performance improvements: strategies to reduce the load on the database by caching our app and taking advantage of the async support to run tasks concurrently with async views to name a few.

However, there is a point where the improvements are not enough anymore:

What if we have tasks to be performed but one or more of them take too long?
- Long-running tasks: when dealing with tasks that might take a long time to complete without affecting the responsiveness of the application.
What if one of the tasks depend on an external API we have no control of?
- Integrating external services: when we want to integrate with external services and need a robust solution to handle communication asynchronous.
What if many things can go wrong and we need to make sure retries are done and any problems are notified?
- Retry mechanism: when we want to implement automatic task retrying in case of failures or temporary errors, improving the task reliability.

These are just a few examples when we need to start thinking out of the box. 📦

It’s time to introduce you to Celery!

Enter Celery…

Celery is an open-source distributed task queue. And what does that mean?

Task can mean a lot of things… In this context, a task generally refers to the smallest unit of work in Celery.

The role of a task queue is to allow us to offload the work - here referred as a task - to another process to be handled asynchronously in the background. Once the task is sent to the queue, no “awaiting” is needed to complete the request/response cycle or proceed to other tasks, thus not blocking the main application flow.

In particular, Celery enables the execution of tasks in a distributed manner, across multiple machines or worker processes, which helps improve the performance and scalability of our applications.

It uses the producer/consumer model. In this model, tasks are generated by the application (producer), sent to a queue (broker) and finally processed by the worker processes (consumer):

In a high-level: when a task is called, our application sends a message to the broker containing the task details. The message is then added the a queue in the broker, awaiting execution. The worker(s) monitors the broker for new tasks and when a worker is free and ready to process a new task, it requests a task from the broker. When the broker receives the request from the worker, it provides it with the next task to be executed.

It’s important to emphasize that the worker executes the task independently from the main application, allowing the main application to proceed with other requests.

The worker(s) can perform operations on the database if needed. Once the task is completed, the worker updates the task status and result. If necessary, the result of the task can be stored.

Backend and Brokers

Brokers serve as a message transport. It enables the communication between producers and consumers. They act as a centralized hub where tasks are stored and can be accessed by the consumers.

At the time of writing, Celery supports RabbitMQ, Redis and Amazon SQS as brokers. Redis performs well for high-speed delivery of small messages. On the flip side, RabbitMQ can handle larger messages than Redis. However, if there is a high influx of messages, you should consider scaling by using Redis or SQS - unless RabbitMQ is running at a very large scale.

Redis and RabbitMQ can also be used as a backend to store the results. Other than that, SQLAlchemy is another option which allows Celery to interface with MySQL, PostgreSQL, SQLite, and more. That’s how Celery can use a SQL database as a result backend.

Due to a lack of resources, Windows is no longer supported since Celery 4.x. In this case, it’s suggested to use other task queues such as huey or Dramatiq.

If you decide to try this out on Windows anyways, share your experience with us on the Fly.io community. I’d love to hear from you!

Since Windows support is not guaranteed, this guide mainly focuses on Unix Systems.

Django ❤️ Celery

Why did we choose Celery?

Since Celery 3.1, Django is supported out of the box without the need for an external library. So, it comes as no surprise that, according to the Django Developers Survey 2022, 35% of the Django users rely on Celery. Besides that, 27% of them consider django-celery one of their favorite third-party Django packages. This package had its last release back in 2016 and it was the old way to integrate Celery with Django.

Buckle up, because Celery is the star ⭐ of today’s article. But first… Let’s uncover our practical application.

Our Application 📰

Let’s say we want to implement an RSS Feed Reader application that aggregates syndicated web content (in which content is made available from one website to other websites) such as we have for this Blog: https://fly.io/django-beats/feed.xml

    <feed xmlns="http://www.w3.org/2005/Atom">
<title>Django Beats</title>
<subtitle>Articles on all things Django from the Team at Fly.io</subtitle>
<id>https://fly.io/</id>
<link href="https://fly.io/"/>
<link href="https://fly.io/django-beats/feed.xml" rel="self"/>
<updated>2023-07-03T00:00:00+00:00</updated>
<author>
<name>Fly</name>
</author>
<entry>
...
</entry>

  

We won’t get into the details of how this works, but Django comes with a high-level syndication-feed-generating framework for creating RSS and Atom feeds, in case you are interested in implementing your own feed.

We can follow or unfollow RSS feeds, access the feeds and manually refresh it to retrieve the latest updates.

When a new feed is added, we want to parse, create and add the feed to our database. Accessing the feed from an external website might take a few seconds - or even minutes ⏳ In addition, the external source might be down! We might need to try a few times to get it right until we realize we need to display an error to the user. In the meantime, we don’t want the user’s browser to be waiting for those tasks to be finished.

That’s a good use-case for taking advantage of a task queue. In this case, we offload the work to fetch and create all the necessary data to another process and free our main process to continue processing other requests. It’s a win-win situation! 🏅

That’s a high-level implementation of our application:

Our Django app serves as a producer and creates a task to be performed when we need to parse a new feed. We’ll use Redis as our broker, where the tasks will be transported. Celery is our consumer, waiting for new tasks to be executed. Those workers fetch the tasks and store the results to our PostgreSQL database.

As mentioned before, there are a few options where we can store the results. For our example, we are storing the results on PostgreSQL to make it easier to visualize the results via the Django Admin - and for our example there is no need in separating the data.

So, shall we start at the beginning?

Setting Up 🛠

Let’s start by installing celery with redis support by using the bundle:

python3 -m pip install "celery[redis]"

celery 5.3.1 and redis 4.6.0 are used in this guide.

This command installs celery and the dependencies for redis in one go to use Redis as a message transport or as a result backend.

Consider a default Django project structure like this:

    - proj/
  - app/
    - __init__.py
    - ...
  - manage.py
  - proj/
    - __init__.py
    - settings.py
    - urls.py

  

For our example, the structure looks like:

    - django-rss-feed-reader/
  - rss_feed/
    - __init__.py
    - ...
  - manage.py
  - rss_feed_reader/
    - __init__.py
    - settings.py
    - urls.py

  

Let’s first define an instance of Celery, here called app. It’s recommended to create a new celery.py file in your project’s folder:

    # <proj>/<proj>/celery.py
# django-rss-feed-reader/rss_feed_reader/celery.py
import os

from celery import Celery

# set the default Django settings module for the 'celery' program
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "rss_feed_reader.settings")

app = Celery("rss_feed_reader")

# all celery-related configuration keys should have a `CELERY_` prefix
app.config_from_object("django.conf:settings", namespace="CELERY")
# load task modules from all registered Django apps
app.autodiscover_tasks()

  

It’s necessary to import this app in your <proj>/__init__.py to ensure we initialized the celery package when Django starts:

    # <proj>/<proj>/__init__.py
# django-rss-feed-reader/rss_feed_reader/__init__.py
from .celery import app as celery_app

__all__ = ("celery_app",)

The basic config is done. Let’s create the task.

In the <app> module where we want to perform the async task, let’s create a new <app>/tasks.py file to implement a very simplified version of our task:

    # <proj>/<app>/tasks.py
# django-rss-feed-reader/rss_feed/tasks.py
import feedparser
from celery import shared_task
from celery.utils.log import get_task_logger
from django.apps import apps

from rss_feed.exceptions import ScrapeRSSFeedFailException

logging = get_task_logger(__name__)


@shared_task(
    bind=True,
    autoretry_for=(ScrapeRSSFeedFailException,),
    retry_backoff=3,
    retry_kwargs={
        "max_retries": 5,
    },
)
def rss_feed_scrape_task(self, feed_id):
    """Scrape and parse the RSS feeds."""
    Feed = apps.get_model("rss_feed", "Feed")
    feed_obj = Feed.objects.get(id=feed_id)

    if self.request.retries > 0:  # Only when retrying
        logging.info(
            "[Task Retry] Attempt %d/%d",
            self.request.retries,
            self.retry_kwargs["max_retries"],
        )
    logging.info("[Started] Scraping data from %s ....", feed_obj.url)
    parsed_feed_dict = feedparser.parse(feed.url)
    if parsed_feed_dict.bozo:
        raise ScrapeRSSFeedFailException(parsed_feed_dict.bozo_exception)
    # ...
    # Code to parse our feed and update "feed_obj"
    # ...
    logging.info("[Finished] Data from %s was scraped.", feed_obj.url)

  

Note that here we need to use the @shared_task decorator instead of @app.task.

Why do we use @shared_task instead of @app.task, you might ask me?

Our app module (also referred as a reusable app) can’t depend on the project itself, where the app instance is instantiated (celery.py). So, it’s not possible to import the app directly. @shared_task is a way to create tasks without having the app instance.

bind=True: when set to True, it turns the task into a bound task method, giving it access to the task instance itself as its first argument (self) - that’s why we have access to self.request.retries and self.retry_kwargs.
autoretry_for=(ScrapeRSSFeedFailException,): specify the exceptions for which the task should be automatically retried if they are raised during task execution. You can use your own custom exceptions.
retry_backoff=3: enable or disable exponential backoff behaviour during task retries. Exponential backoff is a technique used in retry mechanisms where the delay between consecutive retries increases after each attempt. This approach can prevent overwhelming the system with frequent retries and allow potential transient failures to resolve before retrying the task. When it’s set to a number, like 3, the value is used a delay factor: the first retry will delay 3 seconds, the second will delay 6 seconds, the third will delay 12 seconds, the fourth will delay 24 seconds, and so on. If retry_backoff=True it follows the rules of exponential backoff: the first retry will have a delay of 1 second, the second retry will have a delay of 2 seconds, the third will delay 4 seconds, the fourth will delay 8 seconds, and so on. Important: This behaviour only happens if retry_jitter is explicitly set to False. Otherwise, retry_jitter introduces randomness into exponential backoff delays and prevent all tasks in the queue from being executed simultaneously. The delay will be a random number between zero and the delay value calculated by the retry_backoff. By default, retry_jitter=True.
retry_kwargs: used to specify custom arguments for an internal retry.
- "max_retries": 5: the maximum number of retries before giving up. Celery will raise an exception after 5 failed attempts.

These are just a few option, you can find the list of options here.

Now that the setting up us done, let’s start our Django server:

python3 manage.py runserver

Running Redis Locally

If you don’t have Redis installed, you can do so by running:

# For macOS
brew install redis

# For Linux
sudo apt-get install redis

For Windows: Redis is not officially supported on Windows, but you can follow these steps to install Redis for development.

Let’s start the redis process in the background:

For macOS via Homebrew:

brew services start redis

For Linux/Windows:

sudo service redis-server start

By default, the Redis server runs at the address 127.0.0.1 (or localhost) on the TCP port 6379.

Everything is up and running! Let’s prepare our app to use Redis as a broker.

Configuration

Let’s set the environment variable CELERY_BROKER_URL :

    # <proj>/<proj>/settings.py
# django-rss-feed-reader/rss_feed_reader/settings.py
CELERY_BROKER_URL = env.str("REDIS_URL", "redis://localhost:6379/")

  

We are using django-environ to store the configuration to be loaded at runtime. This is a good practice for our deployment. You can read more about it here.

Now that the broker - responsible for the transportation of our tasks - is running, let’s check how we can to spawn the Celery worker in another process.

Running Celery Worker Server Locally

To start our worker in a new process locally, let’s open a new terminal. We can now start our worker server by running:

celery -A rss_feed_reader worker -l info

 -------------- celery@Katias-Air v5.3.1 (emerald-rush)
--- ***** ----- 
-- ******* ---- macOS-13.5-arm64-arm-64bit 2023-07-28 13:20:44
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         rss_feed_reader:0x103b544d0
- ** ---------- .> transport:   redis://localhost:6379//
- ** ---------- .> results:     
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


[tasks]
  . rss_feed.tasks.rss_feed_scrape_task

Let’s break this down…

-A option specify the name of celery instance to use, in our case, rss_feed_reader. To refresh your memory, this was defined in the celery.py file by instantiating our Celery object:

    app = Celery("rss_feed_reader")

  

We are also defining the log level as info using -l. This level determines the severity of the log message generated by the Celery workers. If not set, warning is the default level. If you remember our task, we used logging.info. This will make sure we will see our log output.

There are so many other options you can define, but those basic ones will do it today.

Our app, broker and worker are running! Let’s test it out, first via shell:

python3 manage.py shell

    >>> from rss_feed.models import Feed
>>> from rss_feed.tasks import rss_feed_scrape_task
>>>
>>> feed = Feed(
...     url="https://fly.io/django-beats/feed.xml"
... )
>>> feed.save()
>>>
>>> rss_feed_task = rss_feed_scrape_task.delay(feed.id)
>>> rss_feed_task
<AsyncResult: 18a6d630-e3cf-4580-a10e-f19f64dd394c>
>>> rss_feed_task.status
'SUCCESS'

  

We create a new Feed object and pass the feed.id to the task. At this point, the task runs and we can check the status. All working fine!

Now, let’s simulate raising the exception on autoretry_for:

    >>> from rss_feed.models import Feed
>>> from rss_feed.tasks import rss_feed_scrape_task
>>>
>>> feed = Feed(
...     url="http://feedparser.org/tests/illformed/rss/aaa_illformed.xml"
... )
>>> feed.save()
>>>
>>> rss_feed_task = rss_feed_scrape_task.delay(feed.id)
>>> rss_feed_task
<AsyncResult: d9910c1a-6e45-429d-b3bc-fc435974d51f>
>>> rss_feed_task.status
'RETRY'
>>> # a few seconds later...
>>> rss_feed_task.status
'FAILURE'

  

and evaluate the output:

[2023-08-11 14:26:41,735: INFO/MainProcess] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] received
[2023-08-11 14:26:41,786: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: [Started] Scraping data from http://feedparser.org/tests/illformed/rss/aaa_illformed.xml ....
[2023-08-11 14:26:42,348: ERROR/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: <unknown>:2:0: syntax error
[2023-08-11 14:26:42,355: INFO/MainProcess] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] received
[2023-08-11 14:26:42,366: INFO/ForkPoolWorker-8] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] retry: Retry in 3s: ScrapeRSSFeedFailException(SAXParseException('syntax error'))
[2023-08-11 14:26:45,355: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: [Task Retry] Attempt 1/5
[2023-08-11 14:26:45,370: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: [Started] Scraping data from http://feedparser.org/tests/illformed/rss/aaa_illformed.xml ....
[2023-08-11 14:26:45,931: ERROR/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: <unknown>:2:0: syntax error
[2023-08-11 14:26:45,934: INFO/MainProcess] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] received
[2023-08-11 14:26:45,941: INFO/ForkPoolWorker-8] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] retry: Retry in 0s: ScrapeRSSFeedFailException(SAXParseException('syntax error'))
[2023-08-11 14:26:45,941: INFO/ForkPoolWorker-1] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: [Task Retry] Attempt 2/5
[2023-08-11 14:26:45,954: INFO/ForkPoolWorker-1] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: [Started] Scraping data from http://feedparser.org/tests/illformed/rss/aaa_illformed.xml ....
[2023-08-11 14:26:46,529: ERROR/ForkPoolWorker-1] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: <unknown>:2:0: syntax error
[2023-08-11 14:26:46,533: INFO/MainProcess] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] received
[2023-08-11 14:26:46,541: INFO/ForkPoolWorker-1] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] retry: Retry in 5s: ScrapeRSSFeedFailException(SAXParseException('syntax error'))
[2023-08-11 14:26:51,537: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: [Task Retry] Attempt 3/5
[2023-08-11 14:26:51,554: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: [Started] Scraping data from http://feedparser.org/tests/illformed/rss/aaa_illformed.xml ....
[2023-08-11 14:26:52,108: ERROR/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: <unknown>:2:0: syntax error
[2023-08-11 14:26:52,113: INFO/MainProcess] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] received
[2023-08-11 14:26:52,119: INFO/ForkPoolWorker-8] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] retry: Retry in 20s: ScrapeRSSFeedFailException(SAXParseException('syntax error'))
[2023-08-11 14:27:12,119: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: [Task Retry] Attempt 4/5
[2023-08-11 14:27:12,175: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: [Started] Scraping data from http://feedparser.org/tests/illformed/rss/aaa_illformed.xml ....
[2023-08-11 14:27:12,761: ERROR/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: <unknown>:2:0: syntax error
[2023-08-11 14:27:12,768: INFO/MainProcess] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] received
[2023-08-11 14:27:12,776: INFO/ForkPoolWorker-8] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] retry: Retry in 41s: ScrapeRSSFeedFailException(SAXParseException('syntax error'))
[2023-08-11 14:27:53,768: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: [Task Retry] Attempt 5/5
[2023-08-11 14:27:53,816: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: [Started] Scraping data from http://feedparser.org/tests/illformed/rss/aaa_illformed.xml ....
[2023-08-11 14:27:54,399: ERROR/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f]: <unknown>:2:0: syntax error
[2023-08-11 14:27:54,411: ERROR/ForkPoolWorker-8] Task rss_feed.tasks.rss_feed_scrape_task[d9910c1a-6e45-429d-b3bc-fc435974d51f] raised unexpected: UnpickleableExceptionWrapper('rss_feed.exceptions', 'ScrapeRSSFeedFailException', ("SAXParseException('syntax error')",), "ScrapeRSSFeedFailException(SAXParseException('syntax error'))")
Traceback (most recent call last):
  ...
  File "/.../flyio/django-rss-feed-reader/rss_feed/tasks.py", line 61, in rss_feed_task
    raise ScrapeRSSFeedFailException(exc)
celery.utils.serialization.UnpickleableExceptionWrapper: ScrapeRSSFeedFailException(SAXParseException('syntax error'))

When our custom ScrapeRSSFeedFailException is raised, the process of retries start.

"max_retries": 5 means that 5 retry attempts will be performed before giving up. retry_backoff=3 gives us the delay factor on the exponential backoff behaviour. Since retry_jitter is not explicitly set to False, our delays represent any number between zero and the delay value: 3s, 6s, 12s, 24s, 48s (5 attempts). The actual randomized values were: 3s, 0s, 5s, 20s, 41s.

Cool! The retries work like a charm!

It’s time to check out the interface:

Our app uses a lot of packages that are not discussed in this article like django-htmx, and celery-progress to make the components dynamic without the need of reloading the page. You can find more about django-htmx in this article. We can discuss more about the progress bar in another article.

We add the feed url and click follow. Our view creates the Feed object and the task that will parse the feed in the background. In this case, we implemented a progress bar that checks the task and dynamically display the feed when the task is done.

Note that our app returns a response immediately and a new request can be sent. Multiple tasks are being handled in the background and our page is not blocked or stuck awaiting for the the response. That’s awesome! 🤩

Let’s observe our terminal to see what happened with our worker:

[2023-07-28 13:55:19,540: INFO/MainProcess] Task rss_feed.tasks.rss_feed_scrape_task[e7e42229-544f-45c4-af83-ac7a0d9e1be6] received
[2023-07-28 13:55:19,563: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[e7e42229-544f-45c4-af83-ac7a0d9e1be6]: [Started] Scraping data from https://www.djangoproject.com/rss/weblog/ ....
[2023-07-28 13:55:23,560: INFO/MainProcess] Task rss_feed.tasks.rss_feed_scrape_task[d8dacf6c-8fcc-49cc-96b6-8b724e67960b] received
[2023-07-28 13:55:23,612: INFO/ForkPoolWorker-1] rss_feed.tasks.rss_feed_scrape_task[d8dacf6c-8fcc-49cc-96b6-8b724e67960b][Started] Scraping data from https://fly.io/django-beats/feed.xml ....
[2023-07-28 13:55:24,791: INFO/ForkPoolWorker-8] [Frss_feed.tasks.rss_feed_scrape_task[e7e42229-544f-45c4-af83-ac7a0d9e1be6]: [Finished] Data from https://www.djangoproject.com/rss/weblog/ was scraped.
[2023-07-28 13:55:24,798: INFO/ForkPoolWorker-8] Task rss_feed.tasks.rss_feed_scrape_task[e7e42229-544f-45c4-af83-ac7a0d9e1be6] succeeded in 5.256896165999933s: None
[2023-07-28 13:55:28,089: INFO/ForkPoolWorker-1] [Frss_feed.tasks.rss_feed_scrape_task[d8dacf6c-8fcc-49cc-96b6-8b724e67960b]: [Finished] Data from https://fly.io/django-beats/feed.xml was scraped.
[2023-07-28 13:55:28,100: INFO/ForkPoolWorker-1] Task rss_feed.tasks.rss_feed_scrape_task[d8dacf6c-8fcc-49cc-96b6-8b724e67960b] succeeded in 4.534301334002521s: None

Our main process received the task. Our worker picked up our tasks and ran them asynchronous.

Note that we can now introduce a concept of “prefork pool”. This is a type of worker pool that celery uses to execute tasks concurrently based on Python multiprocessing package. Note that when we started our worker, the concurrency config was displayed:

concurrency: 8 (prefork)

The concurrency refers to the configuration of the worker’s concurrency level and the pool implementation that is being used. You have no idea what I’m talking about? Don’t worry, it’s time to take this apart…

The worker’s concurrency level refers to the number of child processes processing the queue. By default, this is the number of CPUs available on our system, in my case, 8 cores. This means that we will have 8 child processes to handle the tasks in parallel. It’s possible to change this number by using --concurrency argument when starting the worker server:

celery -A rss_feed_reader worker -l info --concurrency 3

The prefork is the default type when no pool type is set via the --pool option.

Some other options are: solo, threads, gevent, eventlet, processes and custom. We won’t cover them in this article but you can find great deep dive resources that you can check after you understand the basics.

prefork works as pre-forking the fixed number of child processes when celery worker starts. When a task needs to be processed, it’s assigned to one of those child processes for execution.

We can verify that by checking our output:

[2023-07-28 13:55:19,563: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[e7e42229-544f-45c4-af83-ac7a0d9e1be6]: [Started] Scraping data from https://www.djangoproject.com/rss/weblog/ ....
[2023-07-28 13:55:23,612: INFO/ForkPoolWorker-1] rss_feed.tasks.rss_feed_scrape_task[d8dacf6c-8fcc-49cc-96b6-8b724e67960b]: [Started] Scraping data from https://fly.io/django-beats/feed.xml ....
[2023-07-28 13:55:24,791: INFO/ForkPoolWorker-8] rss_feed.tasks.rss_feed_scrape_task[e7e42229-544f-45c4-af83-ac7a0d9e1be6]: [Finished] Data from https://www.djangoproject.com/rss/weblog/ was scraped.
[2023-07-28 13:55:24,798: INFO/ForkPoolWorker-8] Task rss_feed.tasks.rss_feed_scrape_task[e7e42229-544f-45c4-af83-ac7a0d9e1be6] succeeded in 5.256896165999933s: None
[2023-07-28 13:55:28,089: INFO/ForkPoolWorker-1] rss_feed.tasks.rss_feed_scrape_task[d8dacf6c-8fcc-49cc-96b6-8b724e67960b]: [Finished] Data from https://fly.io/django-beats/feed.xml was scraped.
[2023-07-28 13:55:28,100: INFO/ForkPoolWorker-1] Task rss_feed.tasks.rss_feed_scrape_task[d8dacf6c-8fcc-49cc-96b6-8b724e67960b] succeeded in 4.534301334002521s: None

Note that ForkPoolWorker-8, one of the child processes, picked the first task and started executing it. Meanwhile, before the first task finished, another child process, ForkPoolWorker-1, picked up the next task and started executing it. First task then finished successfully, and finally our second task succeeded. Those tasks were running in parallel across different child processes. Cool, right? 🤩

From the terminal, we can also check the results of the tasks:

...
[2023-07-28 13:55:24,798: INFO/ForkPoolWorker-8] Task rss_feed.tasks.rss_feed_scrape_task[e7e42229-544f-45c4-af83-ac7a0d9e1be6] succeeded in 5.256896165999933s: None
...
[2023-07-28 13:55:28,100: INFO/ForkPoolWorker-1] Task rss_feed.tasks.rss_feed_scrape_task[d8dacf6c-8fcc-49cc-96b6-8b724e67960b] succeeded in 4.534301334002521s: None

That’s good for quick checks but in production, it might be unfeasible to find and inspect them in this way - kinda impossible when you think about medium/big-sized apps. Can we store those results to easily check them when needed and monitor the execution and performance?

Storing the Results

Yes! 🥳

By default, Celery stores tasks results in-memory, specifically in the memory of the worker process that executed the task, which means that once the worker is restarted or stopped, its results will be lost. Besides that, there’s no shared storage between workers which can be tricky if you need to access results across different workers.

Storing those results enables us to access and query them later.

We can use django-celery-results to archive that. This extension allow us to use either the Django ORM or the Django Cache framework as result backend to store Celery task results in a persistent way.

We can go ahead and install it:

python3 -m pip install django-celery-results==2.5.1

We need to add django_celery_results to INSTALLED_APPS in the settings.py:

    # <proj>/<proj>/settings.py
# django-rss-feed-reader/rss_feed_reader/settings.py
INSTALLED_APPS = [
    ...
    "django_celery_results",  # <-- Updated!
]

  

Create the necessary Celery tables by applying the migrations:

python3 manage.py migrate django_celery_results

And finally, configuring Celery to use the django-celery-results backend:

    # <proj>/<proj>/settings.py
# django-rss-feed-reader/rss_feed_reader/settings.py
CELERY_RESULT_BACKEND = env.str("CELERY_RESULT_BACKEND", "django-db")

  

Note that we are using django-db option for CELERY_RESULT_BACKEND. It uses the Django database as a result backend. The results of our tasks will be stored in the database using Django’s ORM.

It’s also possible to use the cache backend. In this case, we can define:

    # <proj>/<proj>/settings.py
CELERY_CACHE_BACKEND = env.str("CELERY_CACHE_BACKEND", "django-cache")

using the django-cache option.

We can also use the existing cache defined in CACHES setting, for example:

    # <proj>/<proj>/settings.py
CELERY_CACHE_BACKEND = "default"

CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.redis.RedisCache",
        "LOCATION": env.str("REDIS_URL", "redis://localhost:6379/"),
    }
}

  

There are many other options and you can find additional configuration options here.

We can now access those results in the Django Admin:

Everything is ready! Let’s take the next steps and deploy our app to Fly.io!

Deploying to Fly.io 🚀

flyctl is the command-line utility provided by Fly.io.

If not installed yet, follow these instructions, sign up and log in to Fly.io.

Fly Apps

Let’s go ahead and create our app by running:

fly launch

Creating app in /.../flyio/django-rss-feed-reader
Scanning source code
Detected a Django app
? Choose an app name (leave blank to generate one): django-rss-feed
? Select Organization: Fly.io (fly-io)
Some regions require a paid plan (bom, fra, maa).
See https://fly.io/plans to set up a plan.

? Choose a region for deployment: Amsterdam, Netherlands (ams)
App will use ‘ams’ region as primary

Created app ‘django-rss-feed’ in organization ‘fly-io’
Admin URL: https://fly.io/apps/django-rss-feed
Hostname: django-rss-feed.fly.dev
Set secrets on django-rss-feed: SECRET_KEY
? Would you like to set up a Postgresql database now? Yes
? Select configuration: Development - Single node, 1x shared CPU, 256MB RAM, 1GB disk
? Scale single node pg to zero after one hour? Yes
Creating postgres cluster in organization fly-io
Creating app...
Setting secrets on app django-rss-feed-db...
Provisioning 1 of 1 machines with image flyio/postgres-flex:15.3@sha256:c380a6108f9f49609d64e5e83a3117397ca3b5c3202d0bf0996883ec3dbb80c8
Waiting for machine to start...
Machine 784e903f2d76d8 is created
==> Monitoring health checks
  Waiting for 784e903f2d76d8 to become healthy (started, 3/3)

Postgres cluster django-rss-feed-db created
  Username:    postgres
  Password:    <>
  Hostname:    django-rss-feed-db.internal
  Flycast:     fdaa:1:2e2a:0:1::4
  Proxy port:  5432
  Postgres port:  5433
  Connection string: postgres://postgres:<your-internal-postgres-password>@django-rss-feed-db.flycast:5432

Save your credentials in a secure place -- you won’t be able to see them again!

Connect to postgres
Any app within the Fly.io organization can connect to this Postgres using the above connection string

Now that you’ve set up Postgres, here’s what you need to understand: https://fly.io/docs/postgres/getting-started/what-you-should-know/
Checking for existing attachments
Registering attachment
Creating database
Creating user

Postgres cluster django-rss-feed-db is now attached to django-rss-feed
The following secret was added to django-rss-feed:
  DATABASE_URL=postgres://django_rss_feed:<your-postgres-password>@django-rss-feed-db.flycast:5432/django_rss_feed?sslmode=disable
Postgres cluster django-rss-feed-db is now attached to django-rss-feed
? Would you like to set up an Upstash Redis database now? Yes
? Select an Upstash Redis plan Free: 100 MB Max Data Size, ($0 / month)

Your Upstash Redis database django-rss-feed-redis is ready.
Apps in the fly-io org can connect to at redis://default:<your-redis-password>@fly-django-rss-feed-redis.upstash.io
If you have redis-cli installed, use fly redis connect to connect to your database.

Redis database django-rss-feed-redis is set on django-rss-feed as the REDIS_URL environment variable
Wrote config file fly.toml

[INFO] Python 3.11.3 was detected. ‘python:3.11-slim-buster’ image will be set in the Dockerfile.

Validating .../flyio/django-rss-feed-reader/fly.toml
Platform: machines
✓ Configuration is valid

‘STATIC_ROOT’ setting was detected in ‘rss_feed_reader/settings.py’!
Static files will be collected during build time by running ‘python manage.py collectstatic’ on Dockerfile.

Your Django app is ready to deploy!

For detailed documentation, see https://fly.dev/docs/django/

You can find more details about fly launch here and in the docs.

There are a few things worth mentioning at this point:

? Would you like to set up a Postgresql database now? Yes
? Select configuration: Development - Single node, 1x shared CPU, 256MB RAM, 1GB disk
? Scale single node pg to zero after one hour? Yes
Creating postgres cluster in organization fly-io
Creating app...
Setting secrets on app django-rss-feed-db...
Provisioning 1 of 1 machines with image flyio/postgres-flex:15.3@sha256:c380a6108f9f49609d64e5e83a3117397ca3b5c3202d0bf0996883ec3dbb80c8
Waiting for machine to start...
Machine 784e903f2d76d8 is created
....
Postgres cluster django-rss-feed-db is now attached to django-rss-feed
The following secret was added to django-rss-feed:
  DATABASE_URL=postgres://django_rss_feed:<your-postgres-password>@django-rss-feed-db.flycast:5432/django_rss_feed?sslmode=disable
Postgres cluster django-rss-feed-db is now attached to django-rss-feed

A PostgreSQL database is set up and the DATABASE_URL environment variable is added to our app.

? Scale single node pg to zero after one hour? Yes

You will see the option to scale to zero Postgres when creating shared-cpu-1x single node cluster. It’s a great option for hobby projects to save some money 💸 when they are not being used.

? Would you like to set up an Upstash Redis database now? Yes
? Select an Upstash Redis plan Free: 100 MB Max Data Size, ($0 / month)

Your Upstash Redis database django-rss-feed-redis is ready.
Apps in the fly-io org can connect to at redis://default:<your-redis-password>@fly-django-rss-feed-redis.upstash.io
If you have redis-cli installed, use fly redis connect to connect to your database.

Redis database django-rss-feed-redis is set on django-rss-feed as the REDIS_URL environment variable

Fly.io offers a fully-managed, Redis-compatible database service. That’s what we are using for this guide as a broker. The REDIS_URL is automatically set as a secret and used in our settings.py to set CELERY_BROKER_URL:

    # <proj>/<proj>/settings.py
# django-rss-feed-reader/rss_feed_reader/settings.py
CELERY_BROKER_URL = env.str("REDIS_URL", "redis://localhost:6379/")

  

You can check the status, or find the url - in case you didn’t save before - by running:

fly redis status django-rss-feed-redis

Redis
  ID             = KbmqPDlbomNk3iyB6p9                                                                    
  Name           = django-rss-feed-redis                                                                  
  Plan           = Free                                                                                   
  Primary Region = ams                                                                                    
  Read Regions   = None                                                                                   
  Eviction       = Disabled                                                                               
  Private URL    = redis://default:<your-redis-password>@fly-django-rss-feed-redis.upstash.io

If you host your own Redis, you can set this environment variable by running:

fly secrets set REDIS_URL=redis://<your-redis-username>:<your-redis-password>@<your-redis-host>

Besides that, a new fly.toml file was generated and we need to update it accordingly:

    app = "django-rss-feed"
primary_region = "ams"
...

[processes]  # <-- Updated!
  app = "python -m gunicorn --bind :8000 --workers 2 rss_feed_reader.wsgi"
  worker = "python -m celery -A rss_feed_reader worker -l info"

[env]
  PORT = "8000"

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ["app"]  # only applies to the "app" process

  

Let’s understand what’s happening here.

Besides our app, we need to run another process, our worker. To accomplish that, we need to define a [processes] section in the fly.toml. Each process is also referred as a process group. Once we have that, flyctl assumes that all the processes are defined in that section. Adding this section means that our app needs to be added there explicitly. Then we define another process for our worker.

Note that flyctl created http_service section automatically. This section defines a service that listens on ports 80 and 443. It’s a simpler alternative for [[services]] section for apps that only need HTTP or HTTPS services. For now, it’s enough for our use case.

It’s all prepped to be deployed to Fly.io! 🚀

Fly Machines

Fly Machines are lightweight VMs based on Firecracker that start up super fast. Machines are the VM building blocks for the Fly Apps Platform.

It’s time to deploy our app:

fly deploy

...
  Finished deploying

Visit your newly deployed app at https://django-rss-feed.fly.dev/

YAY! 🎉 Our app is up and running:

fly open

Let’s find out more details about the deployment:

fly status

App
  Name     = django-rss-feed                                        
  Owner    = fly-io                                             
  Hostname = django-rss-feed.fly.dev                                
  Image    = django-rss-feed:deployment-01H6P1F1RXGQRE36XJNP9E4JDH  
  Platform = machines                                               

Machines
PROCESS ID              VERSION REGION  STATE   ROLE    CHECKS  LAST UPDATED         
app     6e82933c7d3058  1      ams     started                 2023-07-31T14:01:13Z    
worker  1781194dfdd0e8  1      ams     started                 2023-07-31T13:19:36Z

Both our processes are up and running!

Now, let’s check the logs after triggering a task by using fly logs -a <your-app-name>:

fly logs -a django-rss-feed

...
2023-07-31T14:11:15Z proxy[6e82933c7d3058] ams [info]machine became reachable in 787.641619ms
2023-07-31T14:11:32Z app[1781194dfdd0e8] ams [info][2023-07-31 14:11:32,698: INFO/MainProcess] Task rss_feed.tasks.rss_feed_scrape_task[19168aa2-bab7-472f-8ed6-c89a42865db0] received
2023-07-31T14:11:32Z app[1781194dfdd0e8] ams [info][2023-07-31 14:11:32,711: INFO/ForkPoolWorker-1] rss_feed.tasks.rss_feed_scrape_task[19168aa2-bab7-472f-8ed6-c89a42865db0]: [Started] Scraping data from https://fly.io/django-beats/feed.xml ....
2023-07-31T14:11:37Z app[1781194dfdd0e8] ams [info][2023-07-31 14:11:37,259: INFO/ForkPoolWorker-1] rss_feed.tasks.rss_feed_scrape_task[19168aa2-bab7-472f-8ed6-c89a42865db0]: [Finished] Data from https://fly.io/django-beats/feed.xml was scraped.
2023-07-31T14:11:37Z app[1781194dfdd0e8] ams [info][2023-07-31 14:11:37,263: INFO/ForkPoolWorker-1] Task rss_feed.tasks.rss_feed_scrape_task[19168aa2-bab7-472f-8ed6-c89a42865db0] succeeded in 4.564230344999942s: None

The application used as an example in this article is live here!

📣 If you have a Django-related blog, share your feed with us!

You can also login to the Django Admin using admin/admin. There you can access the task results.

Scaling the Number of Machines

Fly.io allows us to deploy and scale applications (and databases!) all over the world. 🗺 Isn’t that cool?

We can easily scale the number of Machines. Let’s check what we have:

fly regions list

Regions [worker]: ams
Regions [app]: ams

We currently have one Machine for our application and one Machine for our worker, both in ams region.

Besides that, our Redis instance and Postgres database are also running in ams. It’s highly recommended to keep your workers running in the same region as your broker and the result backend. By doing that, we reduce the latency, which means faster communication and task processing, resulting in improved overall performance and responsiveness for your application.

Now, suppose our app is facing a period of high demand. In this case, we might want to process a higher volume of tasks concurrently.

Given the circumstances, we can scale the number of workers, adding a new Machine in Amsterdam, in the Netherlands (ams). We are able to specify the process group (app or worker):

fly scale count worker=2

App 'django-rss-feed' is going to be scaled according to this plan:
  +1 machines for group 'worker' on region 'ams' with size 'shared-cpu-1x'
? Scale app django-rss-feed? Yes
Executing scale plan
  Created 1857299a477d28 group:worker region:ams size:shared-cpu-1x

And verify 2 worker Machines running in ams:

fly scale show

VM Resources for app: django-rss-feed

Groups
NAME    COUNT   KIND    CPUS    MEMORY  REGIONS 
app     1       shared  1       256 MB  ams     
worker  2       shared  1       256 MB  ams(2)

You can also run your apps globally and scale them as you wish, there are many options and other ways to scale your app running on Fly.io!

You should give a try!

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to deploy those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

What’s Next?

Async tasks provides a powerful and flexible solution for handling time-consuming operations, improving the overall performance and responsiveness of web apps. In this article, we explored some key concepts and benefits of using Celery to manage background tasks, deploying to Fly.io! 🗺

This is just the start… here are a few things we can also consider:

Integration in existing projects: evaluate your existing projects and identify resource-intensive tasks that can be offload to Celery workers.
Error handling and monitoring: implement proper error handling and logging mechanisms to track and resolve issues related to the Celery tasks. Use monitoring tools to keep track of task execution and performance.
Task optimization: when it becomes necessary, optimize your tasks for efficient processing. We can analyse and evaluate splitting large tasks into smaller sub-tasks or considering task chaining to improve performance.
Security: Ensure that sensitive data and operations are handle securely within the Celery tasks.
Monitoring and scaling: continuously monitor your Celery workers and task queues to assess performance bottlenecks.

We wish, but this is not a solution to all problems.

It’s important to explore possible solution for your specific use case - you might also want to check “Running tasks concurrently in Django asynchronous views”.

If you have any questions or comments, reach out on the Fly.io Community. That’s a great place to share knowledge, help and get help! 🙂

Organizing Database Queries: Managers vs. QuerySets

2023-07-03T00:00:00+00:00

In this article Mariusz Felisiak introduces concepts of Django’s Managers and QuerySet to organize common database queries. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

I’m always focusing on writing readable and maintainable code. When our project becomes bigger and more complex, the number of models and complexity of data access control can increase significantly. We’re starting to ask ourselves questions: How to organize data structure? How to control access to our data in a reliable way? How to share common logic that protects our data? Where to keep data filters? etc. This article is trying to answer some of these questions by introducing concepts of Django’s Managers and QuerySets. We will describe the biggest differences between them and how they can be used to organize common database queries.

Problem

In all projects there exists logic behind retrieving objects from the database and annotating them. Annotations are really helpful for dealing with calculated/virtual fields (not stored in the database) as they allow moving calculations to the database engine and make them flexible to use. We’re talking about things like:

python3 manage.py shell

    >>> from django.db.models import Value
>>> from django.db.models.functions import Concat
>>> from my_app.models import Person
>>>
>>> persons = Person.objects.annotate(
...     full_name=Concat("first_name", Value(" "), "last_name"),
... )
>>> persons.first().full_name
"Joe Doe"

  

Django doesn’t support virtual fields, but they may be implemented in the near future because Django contributors are actively working on this 🚀.

It’s not uncommon to use the same filters in various places. Django has a concept of Managers and QuerySets which allow us to organize those elements in a reliable way. Let’s explore what they are and how to deal with them 🔎.

Managers and QuerySets

Manager is the interface that defines database operations for our models. Django adds a default manager available through the objects attribute, if we do not define a custom one. As a consequence, each non-abstract model has at least one manager that is a gateway to the model’s data. Have you ever asked yourself what is this objects that we use so often? 🤔 Now you know, it’s a Manager! Django allows us to define multiple managers for the same model. Moreover, methods in the manager can return everything that works for us, they are not limited to the QuerySets. Let’s look at an example:

    # models.py

from datetime import date, timedelta

from django.db import models
from django.db.models.functions import Concat


class PersonManager(models.Manager):
    def with_extra_fields(self):
        return self.annotate(
            full_name=Concat("first_name", models.Value(" "), "last_name"),
        )

    def experienced(self):
        return self.filter(
            join_date__lt=date.today() - timedelta(days=1000)
        )


class PersonAnalyticsManager(models.Manager):
    def number_of_unique_names(self):
        return self.aggregate(
            count=models.Count("last_name", distinct=True),
        )["count"]


class Person(models.Model):
    first_name = models.CharField(max_length=255)
    last_name = models.CharField(max_length=255)
    join_date = models.DateField(default=date.today)

    # We redefine the default manager.
    objects = PersonManager()
    # We add a second manager with analytic methods.
    analytics = PersonAnalyticsManager()

    def __str__(self):
        return f"{self.first_name} {self.last_name}"

  

With managers you can freely share annotations, aggregations, reuse common filters, and more:

python3 manage.py shell

    >>> from .models import Person
>>> Person.objects.with_extra_fields().order_by("full_name")
<QuerySet [<Person: Catherine Smith>, <Person: Joe Doe>, <Person: Omega Smith>]>
>>> Person.objects.experienced()
<PersonQuerySet [<Person: Joe Doe>]>
>>> Person.analytics.number_of_unique_names()
2

  

Unfortunately, there are some inconveniences to using managers. The biggest one is that manager methods cannot be chained 😞, which is really powerful when working with QuerySets.

    >>> Person.objects.experienced().with_extra_fields()
...
AttributeError: 'QuerySet' object has no attribute 'with_extra_fields'

  

How to deal with this? There is a way! We can take the best of both worlds of Managers and QuerySets by defining a custom QuerySet and using it as a manager. For this we will use the as_manager() method. Let’s rewrite our example:

    # models.py

from datetime import date, timedelta

from django.db import models
from django.db.models.functions import Concat


class PersonQuerySet(models.QuerySet):  # Changed to a custom QuerySet.
    def with_extra_fields(self):
        return self.annotate(
            full_name=Concat("first_name", models.Value(" "), "last_name"),
        )

    def experienced(self):
        return self.filter(
            join_date__lt=date.today() - timedelta(days=1000)
        )

    def number_of_unique_names(self):
        return self.aggregate(
            count=models.Count("last_name", distinct=True),
        )["count"]


class Person(models.Model):
    first_name = models.CharField(max_length=255)
    last_name = models.CharField(max_length=255)
    join_date = models.DateField(default=date.today)
    # QuerySet used as a Manager.
    objects = PersonQuerySet.as_manager()

    def __str__(self):
        return f"{self.first_name} {self.last_name}"

  

and check if it works 🥳:

python3 manage.py shell

    >>> Person.objects.experienced()
<PersonQuerySet [<Person: Joe Doe>]>
>>> Person.objects.experienced().with_extra_fields().get().full_name
'Joe Doe'
>>> Person.objects.experienced().number_of_unique_names()
1
>>> Person.objects.number_of_unique_names()
2 

  

In the next section we will cover even more handy usages of custom QuerySets.

Full example

We need an existing or new Django project. Here are some great resources for getting started with Django or deploying your Django app to Fly.io.

With a project ready, let’s get started!

In our example, we’ll use the following models:

    # models.py

from datetime import date

from django.db import models

from .querysets import PersonQuerySet
from .managers import OrderManager


class Person(models.Model):
    first_name = models.CharField(max_length=255)
    last_name = models.CharField(max_length=255)
    join_date = models.DateField(default=datetime.date.today)
    # We redefine the default manager.
    objects = PersonQuerySet.as_manager()


class Order(models.Model):
    seller = models.ForeignKey("Person", on_delete=models.CASCADE)
    sale_date = models.DateField(default=date.today)
    deleted = models.BooleanField(default=False)
    # We redefine the default manager.
    objects = OrderManager()

  

Let’s start by implementing soft-delete for orders.

Soft-delete with a custom Manager

We want to be able to delete orders but at the same time we want to keep historical data to have a track of all orders, including deleted ones. For this, we will use the deleted flag on the Order model. The questions are:

How to inject this feature into our existing project without rewriting all the QuerySets related to orders?
How to make sure deleted orders won’t leak in some places?

Overriding the default manager (objects) is ideal in such cases as it allows us to change the underlying base QuerySet by defining a special method called get_queryset(). This is how we can do this:

    # managers.py

from django.db.models import Manager

class OrderManager(Manager):
    def get_queryset(self):
        return super().get_queryset().filter(deleted=False)

If we want to have backdoor access to all orders in our Django project (including deleted ones), we can define a second manager bypassing this restriction:

    from django.db import models

class Order(models.Model):
    seller = models.ForeignKey("Person", on_delete=models.CASCADE)
    sale_date = models.DateField(default=datetime.date.today)
    deleted = models.BooleanField(default=False)

    objects = OrderManager()
    historical = models.Manager()

  

Let’s find out how it works in practice:

    >>> from order.models import Order, Person
>>> from django.db.models import Count
>>> Order.objects.count()  # Deleted orders are not included.
7
>>> Order.objects.filter(deleted=True).count()
0
>>> Order.historical.count()  # All orders are included.
8
>>> Order.historical.filter(deleted=True).count()
1
>>> # Deleted orders are not included when using related managers.
>>> persons = Person.objects.with_extra_fields().annotate(Count("order"))
>>> for person in persons:
...     print(person.full_name, person.order__count)
... 
Joe Doe 1
Omega Smith 3
Catherine Smith 4

  

Deleted orders will not leak to any QuerySet unless we will explicitly use the historical manager. We can add more methods to our QuerySet and mix them freely as we will see in the next section.

Let’s add one more method to our QuerySet and check out how handy it is to work with different data requirements. Suppose sellers are considered active when they have at least 5 sales in the last 30 days, we can filter them out with the active() method:

    from datetime import date, timedelta

from django.db.models import Count, Value, Q, QuerySet
from django.db.models.functions import Concat


class PersonQuerySet(QuerySet):
    def with_extra_fields(self):
        return self.annotate(
            full_name=Concat("first_name", Value(" "), "last_name"),
        )

    def experienced(self):
        return self.filter(
            join_date__lt=date.today() - timedelta(days=1000)
        )

    def active(self):  # <- Our new method.
        # Sellers with more then 5 orders in the last 30 days are
        # considered active.
        count_from_date = date.today() - timedelta(days=30)
        return self.annotate(
            # Deleted orders are not taken into account because we
            # filtered them out in OrderManager.get_queryset().
            recent_sales_counter=Count(
                "order", filter=Q(order__sale_date__gte=count_from_date)
            ),
        ).filter(recent_sales_counter__gte=5)

    def number_of_unique_names(self):
        return self.aggregate(
            count=Count("last_name", distinct=True),
        )["count"]

  

Let’s give it a spin:

python3 manage.py shell

    >>> from my_app.models import Order, Person
>>> # Active sellers with more then 5 orders in the last 30 days.
>>> Person.objects.active()
<PersonQuerySet [<Person: Joe Doe>, <Person: Catherine Smith>]>
>>> # Active sellers with the number or recent orders, ordered
>>> # by the number of orders. We're chaining 4 QuerySet
>>> # methods, two standard and two custom.
>>> Person.objects.with_extra_fields().active().order_by(
...     "-recent_sales_counter",
... ).values("full_name", "recent_sales_counter")
<PersonQuerySet [
  {'full_name': 'Joe Doe', 'recent_sales_counter': 10},
  {'full_name': 'Catherine Smith', 'recent_sales_counter': 5}
]>
>>> # Active experienced sellers.
>>> Person.objects.with_extra_fields().active().experienced()
<PersonQuerySet [<Person: Joe Doe>]>
>>> # We can exclude experienced sellers.
>>> from django.db.models import Exists, OuterRef
>>> Person.objects.active().exclude(
...     Exists(Person.objects.experienced().filter(pk=OuterRef("pk"))),
... )
<PersonQuerySet [<Person: Catherine Smith>]>
>>> # and so on ... 

  

We can do whatever the heart desires 💃🕺, the Django documentation has many more examples.

Closing Thoughts

Here we made an introduction to extremely useful concepts of Managers and QuerySets in Django that allow us to keep queries under control. Check the Django documentation if you want to dive deeper into all possibilities. If you’ve never used them, it’s time to help your future self and make your work with the ORM smoother, easier to maintain, and more secure!

QuerySets are Lazy

2023-06-28T00:00:00+00:00

In this article we explore the lazy QuerySets in Django. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

Did you know that QuerySets are lazy?

QuerySets have a hidden gem 💎 called lazy evaluation. It is a powerful feature that allows you to postpone the execution of a database’s query until it’s absolutely necessary.

Let’s dive right in.

What are Lazy QuerySets? 💭

When you perform operations on a QuerySet, such as applying a filter, ordering or slicing, Django constructs a query but doesn’t execute it immediately. Instead, the actual database query is executed only when you access the data or explicitly evaluate the QuerySet.

This deferred execution allows for efficient querying and reduces unnecessary database operations.

How do Lazy QuerySets work? 🚀

Let’s say we have a Title model with thousands of records and a Genre model with couple dozens:

This is a simplified version of this project.

    class Genre(models.Model):
    name = models.CharField(max_length=255, unique=True)

    def __str__(self):
        return self.name


class Title(models.Model):
    title = models.CharField(max_length=255)
    genre = models.ManyToManyField(Genre, related_name="titles", blank=True)
    year_start = models.PositiveSmallIntegerField(null=True, blank=True)
    imdb_rating = models.FloatField()

    class Meta:
        ordering = ("-imdb_rating",)

    def __str__(self):
        return self.title

  

This is a simplified version of this project.

By taking advantage of the lazy evaluation, we can efficiently narrow down our results without fetching unnecessary data:

    titles = Title.objects.all()  # Query is not executed yet

# Chaining additional operations: filtering, ordering and slicing
titles = titles.filter(imdb_rating__gte=8.5).order_by("-year_start")[:10]

# Access the data
for title in titles:
    print(f"{title.year_start}: {title.title} [{title.imdb_rating}]")

  

Note that we create the QuerySet fetching all titles.

    titles = Title.objects.all()

  

This operation returns an unevaluated QuerySet.

Subsequently, we filter, order and slice our QuerySet - fetch titles with imdb_rating greater than or equal 8.5, sort in descending order by year_start and return the first 10 titles:

    titles = titles.filter(imdb_rating__gte=8.5).order_by("-year_start")[:10]

  

By this point, no operations to the database were actually executed.

Then, we finally access the data by iterating over the results:

    for title in titles:
    print(f"{title.year_start}: {title.title} [{title.imdb.rating}]")

At this step, the query is finally executed in the database. Note that we ensure that only the filtered, ordered and sliced results are retrieved from the database.

Evaluating the Queries

We can verify locally that the queries aren’t executed yet by checking the SQL statements that Django performed using connection.queries:

python3 manage.py shell

    >>> from catalogue.models import Title
>>> from django.db import connection
>>>
>>> titles = Title.objects.all()
>>> len(connection.queries)
0
>>> titles = titles.filter(imdb_rating__gte=8.5).order_by("-year_start")[:10]
>>> len(connection.queries)
0
>>> for title in titles:
...     print(f"{title.year_start}: {title.title} [{title.imdb_rating}]")
... 
2022: House of the Dragon [8.5]
2019: Parasite [8.5]
2019: Chernobyl [9.4]
2018: Beat Saber [8.9]
2018: Succession [8.9]
2017: Dark [8.7]
2016: Stranger Things [8.7]
2015: Narcos [8.8]
2014: True Detective [8.9]
2013: Vikings [8.5]
>>> len(connection.queries)
1
>>> connection.queries
[
    {
        'sql': '
            SELECT "catalogue_title"."id", "catalogue_title"."title", "catalogue_title"."year_start", "catalogue_title"."imdb_rating" 
            FROM "catalogue_title" 
            WHERE "catalogue_title"."imdb_rating" >= 8.5 
            ORDER BY "catalogue_title"."year_start" 
            DESC LIMIT 10
        ', 
        'time': '0.002'
    }
]

  

We can confirm that only one query was performed and display it by accessing connection.queries.

connection.queries is only available if DEBUG is True.

Exploring the Laziness 🔎

Iterating over a QuerySet is not the only way to trigger an immediate query to the database, some other methods include:

Accessing specific elements: titles[0] or titles.get(pk=1)
Slicing using the step parameter: titles[::2]
Counting: titles.count()
Existence check: titles.exists() or if titles:
Deleting: titles.delete()
Evaluation methods: titles.first(), titles.last(), titles.earliest(), titles.latest(), etc.
Aggregation over a QuerySet: titles.aggregate(Min("year_start"))
Iterating using iterator(): titles.iterator(chunk_size=3000)

Those methods also include the asynchronous versions: aget(), acount(), aexists(), adelete(), afirst(), alast(), alatest(), aearliest(), aaggregate(), aiterator(), etc.

Converting a QuerySet to a list: list(titles)

On the other hand, besides filtering, ordering or slicing, accessing related objects using the double underscore (__) notation, does not immediately fetch the related objects. Instead, they are also lazily fetched when accessed:

    >>> from catalogue.models import Title
>>> from django.db import connection
>>> 
>>> titles = Title.objects.all()
>>> titles = titles.filter(imdb_rating__gte=8.5).order_by("-year_start")
>>> titles = titles.filter(genre__name="Action")  # filter related object
>>> len(connection.queries)
0
>>> for title in titles:
...     print(title.title)
... 
House of the Dragon
The Last of Us
Vikings
Game of Thrones
Inception
The Matrix
>>> len(connection.queries)
1
>>> connection.queries
[
    {
        'sql': '
            SELECT "catalogue_title"."id", "catalogue_title"."title", "catalogue_title"."year_start", "catalogue_title"."imdb_rating" 
            FROM "catalogue_title" 
            INNER JOIN "catalogue_title_genre" 
                ON ("catalogue_title"."id" = "catalogue_title_genre"."title_id") 
            INNER JOIN "catalogue_genre" 
                ON ("catalogue_title_genre"."genre_id" = "catalogue_genre"."id") 
            WHERE ("catalogue_title"."imdb_rating" >= 8.5 AND "catalogue_genre"."name" = \'Action\') 
            ORDER BY "catalogue_title"."year_start" DESC
        ', 
        'time': '0.003'
    }
]

  

The Title model is filtered by genre, which is a ManyToManyField. Since those fields are not explicitly accessed, no additional queries are performed.

Keep in mind that once a query is executed, the retrieved data is stored in memory, and subsequent operation on that QuerySet use the “cached” data rather than querying the database again. This internal caching in memory mechanism is limited to the scope of the QuerySet instance and does not persist beyond the lifetime of that specific QuerySet.

One exception worth mentioning is the iterator(). In order to avoid loading all the objects into memory at once, iterator() fetches a chunk of data directly from the database at a time, without doing any caching at the QuerySet level - take into account that this approach can triggers multiple queries to the database.

This method is particularly ideal for large amount of objects that we only need to access once, resulting in higher performance and reducing the memory usage.

Always evaluate the trade-offs between memory efficiency and the potential for additional database queries.

However, modifying the QuerySet will trigger a new query to the database.

When it comes to QuerySets in Django, understanding which methods trigger an immediate execution and which ones do not, helps us minimizing unnecessary queries and optimize performance.

Wrap-Up 📄

Lazy QuerySets are useful when we need to construct dynamic queries based on runtime conditions or user inputs. It’s also ideal for implementing pagination by retrieving and displaying only a subset of records at a time. Those are just a few examples on how lazy QuerySets might come in handy.

While it offers performance optimizations and flexibility, excessive chaining or complex query construction can make our code harder to read and maintain. It’s always important to keep a balance when applying lazy evaluation to our QuerySets.

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to _deploy_ those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

Caching in Django with Redis

2023-06-19T00:00:00+00:00

In this post we reduce the load on the database and improve the overall performance of our Django app by caching with Redis. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

Web servers are math wizards 🧙🏽‍♂️ that can perform complex calculations to build the final - or partial - response that our user sees. Those calculations can vary from expensive queries to the database to rendering the template.

For most of small/medium-sized apps, building those pages feel instantaneous to users: bellow 100ms response time is our magic 🪄 number. However, ‘with growth usually comes more complexity’. More features, more users, more traffic… that’s when we start thinking about strategies on how to improve the overall performance and user experience of our app. Today, we discuss one of them: caching.

Caching is the process of storing frequently accessed or really expensive data temporarily in cache. Cache refers to a high-speed temporary storage mechanism for this quick access. Think of this as having a data storage closer to the users - and we really like that!

And how does that work in Django, you might ask?

Django ❤️ Redis

Redis - stands for Remote Dictionary Server - is an open-source, in-memory data structure store that can be used as a database, cache, and message broker. It’s known for its high performance, scalability, and versatility, making it a popular choice for many applications.

Redis stores data primarily in memory, allowing for fast read and write operations.

Django 4.0 introduced a built-in support for caching with Redis. The documentation can be found here. As no surprise, Redis is the most popular cache backend the Django developers use, as it’s shown in the Django Developers Survey 2022.

So, that’s what we are choosing to use today. Let’s explore further how we can try Redis locally.

Running Redis Locally

If you don’t have Redis installed, you can do so by running:

# For macOS
brew install redis

# For Linux
sudo apt-get install redis

For Windows: Redis is not officially supported on Windows, but you can follow these steps to install Redis for development.

You can check the documentation here for more details.

For macOS users, you can test your Redis installation by running redis-server:

redis-server

42980:C 14 Jun 2023 12:07:11.089 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
42980:C 14 Jun 2023 12:07:11.089 # Redis version=7.0.9, bits=64, commit=00000000, modified=0, pid=42980, just started
42980:C 14 Jun 2023 12:07:11.089 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
42980:M 14 Jun 2023 12:07:11.089 * Increased maximum number of open files to 10032 (it was originally set to 256).
42980:M 14 Jun 2023 12:07:11.089 * monotonic clock: POSIX clock_gettime
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 7.0.9 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 42980
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           https://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

42980:M 14 Jun 2023 12:07:11.091 * Loading RDB produced by version 7.0.9
42980:M 14 Jun 2023 12:07:11.091 * RDB age 7858749 seconds
42980:M 14 Jun 2023 12:07:11.091 * RDB memory usage when created 1.04 Mb
42980:M 14 Jun 2023 12:07:11.091 * Done loading RDB, keys loaded: 0, keys expired: 0.
42980:M 14 Jun 2023 12:07:11.091 * DB loaded from disk: 0.000 seconds
42980:M 14 Jun 2023 12:07:11.091 * Ready to accept connections

Another alternative is to start/stop the process in the background. Let’s try that:

# For macOS via Homebrew
brew services start redis  # start
brew services stop redis  # stop

# For Linux/Windows
sudo service redis-server start

By default, the Redis server runs at the address 127.0.0.1 (or localhost) with port 6379.

Everything is up and running! Let’s try it out in interactive mode using the Redis CLI:

redis-cli

127.0.0.1:6379> PING
PONG

Caching Our App 🛠

We assume you already have an existing Django application. If not, you can follow the django-imdb we are using as an example for this guide. In this app, we can search our favourite movies, TV shows and games - which are all represented by the Title model.

We are using Django 4.2.2 and redis-py 4.5.5. It’s also recommended to install redis with hiredis support for faster performance, which we will do next:

python3 -m pip install "redis[hiredis]"

python3 -m pip install "redis[hiredis]" explicitly invokes the Python 3.x interpreter. It ensures the use of Python 3.x and it’s recommended, specially if you have both Python 2.x and Python 3.x installed in your system.

The next step is to set up the CACHES setting in our settings.py:

    # settings.py
CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.redis.RedisCache",
        "LOCATION": env.str("REDIS_URL", "redis://localhost:6379/"),
        "KEY_PREFIX": "imdb",
        "TIMEOUT": 60 * 15,  # in seconds: 60 * 15 (15 minutes)
    }
}

  

We are using django-environ to set LOCATION using env.str. You can see more details here.

Our local Redis instance is running on redis://localhost:6379/. django-environ takes environment variables from the .env file. Let’s set the environment variable REDIS_URL in our .env file:

    # .env
REDIS_URL=redis://localhost:6379/

Note that we set TIMEOUT to 15 minutes (900 seconds). If not set, the default value is 5 minutes (300 seconds). If set to None, cache keys never expire. Setting 0 will expire the cache immediately - in other words, “don’t cache”. Set a default value that it’s appropriate for your application.

Optionally, we can check the cache information for our requests using django-debug-toolbar. That’s what we are using here:

python3 -m pip install django-debug-toolbar

Let’s go ahead and add the necessary configuration:

    # settings.py
if DEBUG:
    INTERNAL_IPS = ["127.0.0.1"]  # <-- Updated!

INSTALLED_APPS = [
    # ...
    "django.contrib.staticfiles",
    # ...
    # 3rd party apps
    "debug_toolbar",  # <-- Updated!
]

MIDDLEWARE = [
    "debug_toolbar.middleware.DebugToolbarMiddleware",  # <-- Updated!
    # ...
]

  

The Debug Toolbar is displayed only if our IP address is listed in INTERNAL_IPS - in our case, only if we are in DEBUG mode (DEBUG=True for local development). We should make sure that django.contrib.staticfiles is in our INSTALLED_APPS and also add debug_toolbar. Most of the implementation is done in a middleware, let’s add it to the MIDDLEWARE list as early as possible - the order is important.

Last but not least, add the django-debug-toolbar‘s URL to our project’s urls.py:

    # urls.py
urlpatterns = [
    ...
    path("__debug__/", include("debug_toolbar.urls")),
]

  

Let’s explore some examples on how we can apply that in our django-imdb project’s views.

The Per-View Cache

Title model has the ordering by rating and we want to display the title with the highest rating in our home page. Let’s consider that we will update our database - add new titles and update the ratings for existing titles - every day or even every hour, we can assume that this page won’t change that often, that’s a good candidate for the per-view cache:

    # catalogue/views.py
@cache_page(timeout=60 * 30)  # cache for 30 minutes
def home(request):
    # Display the title with the highest rating in the home page
    title = Title.objects.prefetch_related("genre").first()
    return render(
        request=request,
        template_name="home.html",  # base template
        context={"title": title},
    )

  

The per-view will cache the output of individual views.

We add the @cache_page decorator to our view and customize the timeout (60 * 30).

If you configured the django-debug-tollbar, you can check the information in the panel. Once you access the home page for the first time, two queries are executed in our database to fetch the highest rating title, you can check that on SQL panel:

If we move to the Cache panel, we notice that this request cached two components: cache_header and cache_page. As the name already implies, the first one caches the headers in the HTTP response and the second, caches the content of the view itself.

Let’s refresh the page and observe what happens:

Ta-da! 🥳 No queries were performed in the database!

And we can verify that our cached data was retrieved, performing two get calls to fetch the header and page content.

Alternatively, we can retrieve all the keys with the KEY_PREFIX prefix - for our example, imdb - we defined in the settings.py using redis-cli:

redis-cli

127.0.0.1:6379> keys imdb:*
1) "imdb:1:views.decorators.cache.cache_header..592d38382.en-us.UTC"
2) "imdb:1:views.decorators.cache.cache_page..GET.592d38382.d4cf8427e.en-us.UTC"

If you prefer using a Graphic User Interface (GUI), you can check RedisInsight out.

There are other caching levels you can explore such as per-site cache, and template fragment caching which we won’t cover in this article. Check them out and see what fits best in your case.

Now that we’re are familiar with the workings, let’s take our search view into the next example. For most situations, the results for an specific search value can change - new titles can be added, updated or removed. But most apps like ours do not require real-time updates.

Per-view cache is effective for static or semi-static views that don’t depend on user-specific or dynamic data - like our home page. Sometimes, caching the entire page can be excessive. So, how can we leverage the use of caching for dynamic views like our search page?

The Low-Level Cache API

The low-level cache API provides more fine-grained control over caching. It allow us to cache specific function calls, database queries - yeah, those expensive ones! -, or even fragments of our template.

This approach is suitable when we need more control to cache reusable components or dynamic data and it’s beneficial when dealing with more complex views.

That’s one way how we could apply it to our search view, caching the queryset results and avoiding hitting our database:

    # catalogue/views.py
def search(request):
    query = request.GET.get("q", "")
    page_num = request.GET.get("page", 1)

    cache_key = f"search_{query}" if query else "search_all"
    titles = cache.get(key=cache_key)
    if titles is None:
        titles = Title.objects.prefetch_related("genre")
        if query:
            titles = titles.filter(title__icontains=query)
        cache.set(
            key=cache_key,
            value=titles,
            timeout=60 * 15,  # in seconds (900s or 15min)
        )
    page = Paginator(object_list=titles, per_page=5).get_page(page_num)

    return render(request=request, template_name="search.html", context={"page": page})

  

Let’s break this code down.

    cache_key = f"search_{query}" if query else "search_all"
titles = cache.get(key=cache_key)

cache.get() returns None if the object with the specific key doesn’t exist in cache. in this scenario, we query the database and then save the result of the queryset in cache, so the next search with the same query term will fetch results from the cache:

    if titles is None:
    titles = Title.objects.prefetch_related("genre")
    if query:
        titles = titles.filter(title__icontains=query)
    cache.set(
        key=cache_key,
        value=titles,
        timeout=60 * 15,  # in seconds (900s or 15min)
    )

  

For demonstration purposes, we are setting a new timeout of 15 minutes (900 seconds). Keep the timeout value in mind and set according to your use case.

The low-level cache API can store objects in cache with any level of granularity. We can cache any Python objects that can be pickled 🥒 safely: strings, dictionaries and lists of model objects are just a few examples.

Cool! Everything is working on my local machine. How do I deploy my app to Fly.io?

Deploying to Fly.io 🚀

flyctl is the command-line utility provided by Fly.io.

If not installed yet, follow these instructions, sign up and log in to Fly.io.

Before the deployment, we need to create the Redis instance for our production environment.

Upstash Redis

Fly.io offers a fully-managed, Redis-compatible database service. That’s what we are using for this guide. Let’s go ahead and create it:

fly redis create

? Select Organization: django-imdb (django-imdb)
? Choose a Redis database name (leave blank to generate one): django-imdb-redis
? Choose a primary region (can't be changed later) Amsterdam, Netherlands (ams)

Upstash Redis can evict objects when memory is full. This is useful when caching in Redis. This setting can be changed later.
Learn more at https://fly.io/docs/reference/redis/#memory-limits-and-object-eviction-policies
? Would you like to enable eviction? No
Some regions require a paid plan (fra, maa).
See https://fly.io/plans to set up a plan.

? Optionally, choose one or more replica regions (can be changed later):
? Select an Upstash Redis plan Free: 100 MB Max Data Size

Your Upstash Redis database django-imdb-redis is ready.
Apps in the django-imdb org can connect to at redis://default:<your-redis-password>@django-imdb-redis.upstash.io
If you have redis-cli installed, use fly redis connect to connect to your database.

Once provisioned, the primary region can’t be changed, so be alert!

If you plan to deploy in a single region, ensure that your database is deployed in the same region as your application.

Once, created, you can check the list of Redis databases:

fly redis list

NAME                    ORG             PLAN    EVICTION        PRIMARY REGION  READ REGIONS
django-imdb-redis       django-imdb     Free    Disabled        ams

You can also check more detailed information, including the Redis URL in case you didn’t take note:

fly redis status django-imdb-redis

Redis
  ID             = MA8qB9aAn8oLVHZ8qAw
  Name           = django-imdb-redis
  Plan           = Free
  Primary Region = ams
  Read Regions   = None
  Eviction       = Disabled
  Private URL    = redis://default:<your-redis-password>@django-imdb-redis.upstash.io

By connecting a database, you can run Redis commands on redis-cli:

fly redis connect

? Select a database to connect to django-imdb-redis (ams) django-imdb
Proxying local port 16379 to remote [fdaa:1:8519:0:1::3]:6379
127.0.0.1:16379> set foo bar
OK
127.0.0.1:16379> get foo
"bar"
127.0.0.1:16379>

It’s possible to access the Upstash web console by running:

fly redis dashboard <your-org-slug>

Secrets

Now that we got the Redis URL, we can set the environment variables to be used in our production deployment. Starting with REDIS_URL:

    fly secrets set REDIS_URL=redis://default:<your-redis-password>@django-imdb-redis.upstash.io

  

Or wherever your Redis server is running:

    fly secrets set REDIS_URL=redis://<your-redis-username>:<your-redis-password>@<your-redis-host>

  

Everything is prepped! Let’s deploy our application.

`fly deploy`

It’s time! 🚀 We can run:

fly deploy

...
Running django-imdb release_command: python manage.py migrate
  release_command e2865079c61e08 completed successfully
Updating existing machines in 'django-imdb' with rolling strategy
  [1/2] Machine e784e903fd9078 [app] update finished: success
  [2/2] Machine 5683076c791e28 [app] update finished: success
  Finished deploying

Visit your newly deployed app at https://django-imdb.fly.dev/

Our app is now running and caching data! ⚙️ We can check it out:

fly open

Our app is running on https://django-imdb.fly.dev/ if you want to check the final results!

What’s Next?

There are multiple factors that can affect performance of our Django apps. Caching is just one of the strategies but as any other strategy, it doesn’t fit all cases. We should be aware of what to cache and what not. Remember to carefully consider what data to cache and set appropriate timeouts to ensure freshness of the cached data.

This isn’t a solution, it’s a process! It’s important to monitor 👀 our application and adjust our caching strategy along the way! Explore frequent access data and expensive operations, and start from there.

Remember: not everything is cache-worthy!

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to _deploy_ those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

DRY: Template Rendering with Context Processors

2023-05-11T00:00:00+00:00

Mariusz Felisiak, a Django and Python contributor and a Django Fellow, explores how to simplify template rendering with context processors. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

Django has many hidden gems that are neither widely used nor well known. In this article, I will discuss context processors which are one of my favorites. They allow following the DRY (don’t repeat yourself) principle in template rendering and keep your code more maintainable. Let’s check how it works and why it’s worth using.

Let’s get some context

Context is a dictionary-like object that a template renders together with a request. In most cases we pass it as a standard dictionary to the render() shortcut. For example:

    # views.py

from django.shortcuts import render

def my_hello_view(request):
    greetings = "Welcome aboard ⛵"
    return render(request, "my_app/hello.html", context={"greetings": greetings})

<!-- hello.html -->

{{ greetings }}

So far, so good, but what if we have information that should be available for every template. How do we inject them without rewriting everything and repeating ourselves? 🤔

Context processors are helpers designed for exactly this. They are functions that accept a request and return a dictionary that is populated to the template context when any template is rendered. See the following stub:

    def my_context_processor(request):
    ...
    return {"key": value}

  

The list of used context processors is controlled by the "context_processors" option of the TEMPLATES setting.

Django includes many built-in context processors, but we can also implement our own. To put it all together, let’s see a full example.

Full example

We need an existing or new Django project. Here are some great resources for getting started with Django or deploying your Django app to Fly.io.

With a project ready, let’s get started!

Let’s assume that we want to share information about the current version and environment of our project for all templates. Environment can be defined as a setting:

    # my_project/settings.py

ENVIRONMENT = "staging"

and version is stored in the project:

    # my_project/__init__.py

VERSION = (2, 23, 1, "rc", 2)

__version__ = ".".join(str(part) for part in VERSION)

We can define a new context processor called metadata() that will add both information to the template context:

    # my_project/context_processors.py

from django.conf import settings

from . import __version__

def metadata(request):
    return {
        "version": __version__,
        "environment": settings.ENVIRONMENT,
    }

  

Then we add the dotted Python path to it ("my_project.context_processors.metadata") to the list of context processors:

    # my_project/settings.py

...

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
                # Our context processor:
                "my_project.context_processors.metadata",
            ],
        },
    },
]

  

That’s it! Information about version and environment are now available for all templates 🥳

We can use them, for example, to display a header about the current version:

    <!-- templates/main.html -->

<!DOCTYPE html>
<html lang="en" dir="ltr">
  <head>
    ...
    <style>
        .version-header {
            text-align: center;
            font-size: 1.3em;
            font-weight: 400;
            padding: 20px;
        }
        .staging-header {
            background-color: yellow;
            color: black;
        }
        .development-header {
            background-color: green;
            color: white;
        }
    </style>
  </head>
  <body>
    {% if environment != "production" %}
        <div class="{{environment}}-header version-header">
            {{ environment|capfirst }} environment, version {{ version }}.
        </div>
    {% endif %}
    ...
  </body>
</html>

  

Believe me, it has saved me many times from making a mistake on a production environment. This is what it looks like in the development version when the ENVIRONMENT setting is set to "development":

and this is how it appears on the staging when the ENVIRONMENT setting is set to "staging":

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to _deploy_ those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

Closing thoughts

Context processors are really handy for sharing information between templates and I believe they should be used more widely.

Built-in context processors mainly provide different settings to the templates, but there are more sophisticated uses, such as providing an instance of currently logged-in user with its permission by django.contrib.auth.context_processors.auth, or passing token needed by CSRF (Cross Site Request Forgery) protection by django.template.context_processors.csrf. Sky’s the limit 🚀

Do you already have an idea how to use them in you project? 💡 Try them and share!

Running tasks concurrently in Django asynchronous views

2023-04-13T00:00:00+00:00

Mariusz Felisiak, a Django and Python contributor and a Django Fellow, explores how to use recent async improvements in Django to run multiple async tasks in an asynchronous view! Django on Fly.io is pretty sweet. Check out how you can be up and running on Fly.io in just minutes.

Async support has really been improving and expanding in Django! Since Django 3.0 with the addition of ASGI support (Asynchronous Server Gateway Interface) there has been a steady march of improvements that bring Django closer to having a full asynchronous request-response cycle.

Now we’re to the point where there’s enough foundational support that interesting things are possible for our “normal web apps.”. This is where it gets really exciting for people! Here we’ll take a look at how we can start using async views with Django.

Reviewing async code can be challenging. So, together, we’ll walk through creating our first asynchronous view in Django.

Buckle-up for our async-journey together!

Brief history of async support

The brief timeline for adding async support to Django shows it’s been careful, steady, and intentional.

Django 3.0 → ASGI support
Django 3.1 → Asynchronous views and middlewares
Django 4.0 → Asynchronous API for cache backends
Django 4.1 → Asynchronous ORM interface
Django 4.2 →Asynchronous streaming responses, asynchronous Model and related manager interfaces, and psycopg version 3 support which provides asynchronous connections and cursors
Django 5.0 (currently under development 🏗️) → Asynchronous signal dispatch, coroutine request.auser(), and more (?)

Seeing the trend and what’s under development is exciting!

Problem

With all the excitement around asynchronous web development, you’d like to get some of those benefits in your own Django app.

These are the questions we’re setting out to answer:

How do we write an asynchronous view in Django?
What can be done asynchronously in Django?
How to use Django with ASGI (Asynchronous Server Gateway Interface) servers?

Solution

We need an existing or new Django project. Here are some great resources for getting started with Django or deploying your Django app to Fly.io.

With a project ready, let’s get started!

How to write an asynchronous view?

First, an asynchronous view is a coroutine function defined with the async def syntax that accepts a request and returns a response. We can use it directly in a URL configuration as any other standard view.

This is what it looks like:

    # urls.py
from django.http import HttpResponse
from django.urls import path


async def my_view(request):
    ...
    return HttpResponse(...)


urlpatterns = [
    ...
    path('my_view/', my_view),
]

  

Django automatically detects async views and runs them in an async context, so we don’t have to do anything else to make them work! These are also supported under ASGI and WSGI mode. However, Django emulates ASGI style when running async views under WSGI, and this kind of context-switching causes a performance penalty. That’s why it’s more efficient to run async views under ASGI.

We have many functionalities that can be asynchronous or at least async-compatible, and as such can be used in async views. If we want to use part of our code or part of Django that is still synchronous, we can wrap it with asgiref.sync.sync_to_async() from asgiref package (asgiref is mandatory for any Django installation):

    # views.py
from asgiref.sync import sync_to_async


@sync_to_async
def my_sync_function():
    ...


async def my_view(request):
    ...
    result = await my_sync_function()
    ...
    return HttpResponse(...)

  

Notice how our synchronous function my_sync_function() is decorated with @sync_to_async. This creates a bridge between the sync and async contexts, allowing us to use synchronous functions in async views.

Example

Let’s write an asynchronous view together with some of available async options. For this example, we have a Django project for gathering information about open source contributors. The first step for a new user is providing an email address. Our view should do a few things:

check if an email is not already registered,
download an avatar,
return a web response.

There’s no reason we cannot perform the first two tasks separately. They don’t rely on each other. We can’t finish our render until both are done, so this is a great opportunity to let those tasks run concurrently and speed up our overall response!

Our view will cover asynchronous examples for:

using the ORM, and
calling an external API.

Both are I/O bound tasks and the real work is being done outside, so they are great tasks for being asynchronous.

Using the ORM

First, we define a coroutine to check if an email is not already registered. QuerySet has an asynchronous interface for all data access operations. Methods are named as synchronous operations but with an a prefix (this is a general rule for creating async variants in Django). We’ll use QuerySet.aexists(), that is an async version of .exists(), and filter against an email address:

    # helpers.py
from django.contrib.auth import get_user_model

async def is_email_registered(email):
    return await get_user_model().objects.filter(email=email).aexists()

Unfortunately, the underlying database operation is synchronous because it uses the sync_to_async() wrapper and a synchronous connection (as asynchronous database drivers are not yet integrated, or even exist for most databases).

For Django 4.2+, when using newly introduced psycopg version 3 support and a PostgreSQL database you can make it fully asynchronous! It takes some effort, as you have to initialize a new connection and perform a raw SQL statement, but it’s possible to do this fully asynchronously. This is how we have to do it until async connections are not supported in the Django ORM:

    # helpers.py
import psycopg

from django.contrib.auth import get_user_model
from django.db import connection

async def is_email_registered(email):
    # Find and quote a database table name for a Model with users.
    user_db_table = connection.ops.quote_name(get_user_model()._meta.db_table)
    # Create a new async connection.
    aconnection = await psycopg.AsyncConnection.connect(
        **{
            **connection.get_connection_params(),
            "cursor_factory": psycopg.AsyncCursor,
        },
    )
    async with aconnection:
        # Create a new async cursor and execute a query.
        async with aconnection.cursor() as cursor:
            await cursor.execute(
                f'SELECT TRUE FROM {user_db_table} WHERE "email" = %s',
                [email],
            )
            row = await cursor.fetchone()
            return row[0] if row else False

  

We have a database operation that can happen asynchronously! Now, let’s turn our focus to the next asynchronous task we want perform, which is the API call.

Calling external API

Let’s start by defining an async function to fetch an avatar from an external URL. The httpx package is a great solution for this as it defines AsyncClient that provides async methods for all types of requests:

    # helpers.py
import base64
import hashlib

import httpx

async def get_gravatar(email):
    # URL with the avatar for the given email address
    # see https://gravatar.com/site/implement/images/.
    gravatar_hash = hashlib.md5(
        email.lower().strip().encode(),
        usedforsecurity=False,
    ).hexdigest()
    gravatar_url = f"https://www.gravatar.com/avatar/{gravatar_hash}.png"

    # Make HTTP GET request asynchronously.
    async with httpx.AsyncClient() as client:
        response = await client.get(gravatar_url)

    if response.status_code == httpx.codes.OK:
        # Return an avatar encoded with base64.
        return base64.b64encode(response.content).decode()
    return None

  

Awesome! With our external API task written, we’re ready to try running both of them concurrently.

Running coroutines concurrently

Now, we have all steps covered by coroutine functions and we can gather them together in an asynchronous view new_contributor():

    # forms.py
from django import forms

class NewContributorForm(forms.Form):
    email = forms.EmailField(required=True, label="Email address:")

    # views.py
from django.shortcuts import render

from .forms import NewContributorForm
from .helpers import get_gravatar, is_email_registered


async def new_contributor(request):
    if request.method == "POST":
        form = NewContributorForm(request.POST)
        if form.is_valid():
            email = form.cleaned_data["email"]
            # Check if the given email address is already registered.
            is_registered = await is_email_registered(email)
            # Fetch an avatar.
            avatar = await get_gravatar(email)
            # Render the second registration step.
            return render(
                request,
                "registration/new_contributor_step_2.html",
                {
                    "is_registered": is_registered,
                    "email": email,
                    "avatar": avatar,
                },
            )
    else:
        # Return an empty form.
        form = NewContributorForm()
    return render(request, "registration/new_contributor.html", {"form": form})

  

The above view has an important disadvantage, it does not run coroutines concurrently. To do this we can use:

asyncio.gather() which runs awaitables concurrently and returns an aggregated list of values when all have succeeded:

    is_registered, avatar = await asyncio.gather(
    is_email_registered(email),
    get_gravatar(email),
)

  

or we can do this in a more modern way with asyncio.TaskGroup() (available in Python 3.11+) where all tasks are awaited when the context manager exits:

    async with asyncio.TaskGroup() as task_group:
    task1 = task_group.create_task(is_email_registered(email))
    task2 = task_group.create_task(get_gravatar(email))
is_registered = task1.result()
avatar = task2.result()

  

Our final asynchronous view 🎉:

    # views.py
import asyncio

from django.shortcuts import render

from .forms import NewContributorForm
from .helpers import get_gravatar, is_email_registered

async def new_contributor(request):
    if request.method == "POST":
        form = NewContributorForm(request.POST)
        if form.is_valid():
            email = form.cleaned_data["email"]
            async with asyncio.TaskGroup() as task_group:
                task1 = task_group.create_task(is_email_registered(email))
                task2 = task_group.create_task(get_gravatar(email))
            return render(
                request,
                "registration/new_contributor_step_2.html",
                {
                    "is_registered": task1.result(),
                    "email": email,
                    "avatar": task2.result(),
                },
            )
    else:
        form = NewContributorForm()
    return render(request, "registration/new_contributor.html", {"form": form})

  

Let’s take a look how it works:

We did it! When our users enter their email address, we perform two separate and concurrent tasks. One task uses the email to fetch an avatar from an external service through an API call. The other task does a database search.

When both tasks are done, we finish rendering the page.

Let’s check the options for deploying our project with ASGI.

ASGI servers options

Django supports deploying with ASGI by creating an entry-point (<your_project>/asgi.py file) with an application callable for ASGI web servers. The official Django documentation contains details how to deploy with the following servers:

Personally, I prefer daphne as it provides the built-in runserver command that allows to run your project under ASGI during development.

Closing thoughts

There are of course many other use cases for asynchronous stuff in Django 🔥, here we touched only the tip of the iceberg! Performing standalone I/O bound tasks are great for being asynchronous, we can highlight here:

using a database,
making an API call,
reading a file (see aiofiles), or
executing external shell commands, etc.

The first two of which are discussed in this article.

Async support is getting broader with each new version of Django, but it’s pretty mature already 🧑. You no longer need to leave your favorite framework if you have (or simply want) to use asynchronous code.

Async-world is dynamically developed both in Python and Django, so it’s time give it a try.

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to _deploy_ those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

A 'No JS' Solution for Dynamic Search in Django

2023-03-13T00:00:00+00:00

In this post we take advantage of HTMX requests to do partial rendering for list views in Django. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

Django is one of the most used server-side frameworks out there. It uses MTV (Model-Template-View) design pattern to build highly scalable and maintainable apps.

Even though Django is a very versatile framework, one of the things that annoys me the most is the fact that - for a minimal Django setup - it reloads the entire page to get a response. What if we could render individual parts of the HTML page instead of having to reload everything?

Fortunately, there are ways to accomplish that.

The first straightforward option we can think of is to use Javascript (JS). But since we all love Django and probably want to avoid having to write some JS code, I’d like to share with you another way: HTMX!

HTMX is a library created to allow us to use modern browser features - like partial rendering - directly from our HTML, rather than using Javascript. Cool, right? That’s what we are looking for.

Let’s see an example of what we’re shooting for. Check this out:

When the user types a few letters and pauses, it automatically runs the search and updates the search results. No full-page refresh used. The user’s cursor and search text remains and they can add more text to search further. The rest of this post covers how we can achieve that.

💾 You can find the Github repo used in this guide here to follow along with the article.

Django’s Good Old Full Page Reload

We have a Django app that we can search for our favourite movies, TV shows and games - named Django IMDb. We are using some data from the OMDb, the Open Movie Database API. The app lists the titles based on our search by pressing Enter key. Our app requires a full page reload to display the results for both search functionality and pagination.

Here we have our search view with pagination:

Let’s take a look into our current code.

To simplify, we define a simple search function based view in views.py:

    # views.py
def search(request):
    search = request.GET.get('q')
    page_num = request.GET.get('page', 1)

    if search:
        titles = Title.objects.filter(title__icontains=search)
    else:
        titles = Title.objects.none()
    page = Paginator(object_list=titles, per_page=5).get_page(page_num)

    return render(
        request=request,
        template_name='search.html',
        context={
            'page': page
        }
    )

  

Our search view receives the parameter q that represents our search value and the page parameter. We are filtering titles that contain the search value q and getting the specific page if page is set, otherwise, we get the first page. We are using a Paginator which will facilitate the pagination in our template. Our page object contains the list of titles.

We also define the search url in urls.py:

    # catalogue/urls.py
from django.urls import path
from . import views

urlpatterns = [
    path('search/', views.search, name='search'),
]

  

In our search.html page, we add our <form> tag with an <input> for our search:

    <!-- search.html -->
<div class="search">
    <form action="{% url 'search' %}" class="form">
        <input name="q"
               value="{{ request.GET.q }}"
               placeholder="Search for a title"
        >
        </p>
    </form>
</div>

  

In the form, we specify the action attribute, which will call our search/ url when Enter key is pressed. Unless explicitly specified, the default method is GET. To keep the search value in the input field after reloading the page, we set value="{{ request.GET.q }}".

In the same page, we also have our list to be displayed and a pagination:

    <!-- search.html -->
<section id="results">
  <div class="results">
      {% for title in page.object_list %}
          <div class="result">
            <!-- display fields -->
            ...
          </div>
      {% endfor %}
  </div>
  <div class="pagination">
      {% if page %}
          {% if page.number != 1 %}
              <a class="page first-page" href="?q={{ request.GET.q }}&page=1">
                &laquo; First
              </a>
          {% endif %}
          {% if page.has_previous %}
              <a class="page" href="?q={{ request.GET.q }}&page={{ page.previous_page_number }}">
                {{ page.previous_page_number }}
              </a>
          {% endif %}
          <span class="page">
            {{ page.number }}
          </span>
          {% if page.has_next %}
              <a  class="page" href="?q={{ request.GET.q }}&page={{ page.next_page_number }}">
                {{ page.next_page_number }}
              </a>
          {% endif %}
          {% if page.number != page.paginator.num_pages %}
              <a  class="page last-page" href="?q={{ request.GET.q }}&page={{ page.paginator.num_pages }}">
                &raquo; Last
              </a>
          {% endif %}
      {% endif %}
  </div>
</section>

  

Now, let’s take the next steps and transform our search page!

HTMX to the Rescue! 🚀

We want to add dynamic functionality to our page without requiring a full page reload. To accomplish that, we will use one of the most popular ways today: HTMX. This library gives us access to AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, through attributes. If that techno soup sounds like a lot, don’t worry! Essentially, it means we can create really cool dynamic pages using the Django we love and not needing to turn to a whole other JavaScript tool-chain and framework!

We will go over some of the most common attributes in this article and show what are they used for and how to use them.

In our example, we use the django-htmx package. Let’s take a look into it.

`django-htmx`

The django-htmx package was created by Adam Johnson, one of the members of the Django Project Technical Board. django-htmx provides us with extensions for using Django with htmx. Let’s go ahead and installed it using pip:

    python3 -m pip install django-htmx==1.14.0

  

With the package installed, let’s add it to the INSTALLED_APPS in our settings.py:

    # settings.py
INSTALLED_APPS = [
    ...
    # 3rd party apps
    'django_htmx',
]

  

And add the HtmxMiddleware to the MIDDLEWARE:

    # settings.py
MIDDLEWARE = [
    ...
    'django_htmx.middleware.HtmxMiddleware',
]

  

The middleware makes request.htmx available in our view, which allows us to switch behavior for HTMX type requests. We will use that to distinguish the requests in our view.

django-htmx does not include htmx itself. You can download htmx.min.js from it’s latest release. After that, add the Javascript file in your static directory - for our example, static/js folder - and reference it in your base template, within the <head> tag:

    <!-- base.html -->
{% load static %}
<head>
    ...
    <!-- Javascript -->
    <script type="text/javascript" src="{% static 'js/htmx.min.js' %}" defer></script>
</head>

  

Note that we set the defer attribute. This attribute specifies that the downloading happens in the background while parsing the rest of the page.

Partial Search View

Let’s define our partial_search view in views.py:

    # views.py
def partial_search(request):
    if request.htmx:
      search = request.GET.get('q')
      page_num = request.GET.get('page', 1)

      if search:
          titles = Title.objects.filter(title__icontains=search)
      else:
          titles = Title.objects.none()
      page = Paginator(object_list=titles, per_page=5).get_page(page_num)

      return render(
          request=request,
          template_name='partial_results.html',
          context={
              'page': page
          }
      )
    return render(request, 'partial_search.html')

  

As mentioned before, request.htmx is available in our view. We use it to decide what will be performed. If the request is made with htmx, the partial_results.html will be used, otherwise, we render partial_search.html.

With that done, let’s add our new url:

    # catalogue/urls.py
urlpatterns = [
    path('partial-search/', views.partial_search, name='partial_search'),
]

  

Let’s check how those templates partial_search.html and partial_results.html look.

🔍 Dynamic Search

We will start with the partial_search.html. Our <form> is removed given that our <input> can now trigger events. Since our page will not be reloaded, we can remove value="{{ request.GET.q }}" from the input, we don’t need it anymore.

    <!-- partial_search.html -->
<input name="q"
       placeholder="Search for a title"
       hx-get="{% url 'partial_search' %}"
       hx-target="#results"
       hx-trigger="input delay:0.2s"
>
...
<section id="results">
    <div class="results">
        {% include 'partial_results.html' %}
    </div>
</section>

  

A few htmx attributes (hx-*) are added to the element, let’s take a look into each of them:

hx-get: issue a GET request to the specific URL.
- Issue a GET request to our partial-search/ url.
hx-target: specifies a target element for swapping - if not specified, it’s the element itself.
- We define the <section id="results"> as the target to be swapped - #results means is the unique element with id="results". Since hx-swap is not defined, the default is set to innerHTML, which replaces everything inside the target element, <section id="results">...</section>.
hx-trigger: specifies the event that triggers the request.
- HTMX supports multiple triggers separated by comma. Standard events can have modifiers, e.g. input delay:0.2s. The standard event is input of the <input> field and there will be a delay of 0.2 seconds before the event is triggered. This is just an example, customize as needed!

Now, let’s see how to replace the pagination by a Load More button.

🪄 Load More

Let’s check out another way we can leverage htmx in our website, by replacing the usual pagination with a Load More button. The approach behind this button is to render additional content to the page when clicked, without reloading the entire page.

    <!-- partial_results.html -->
{% for title in page.object_list %}
    <div class="result">
        ...
    </div>
{% endfor %}
{% if page %}
    <div id="load-more">
        {% if page.has_next %}
            <div class="load-more">
                <button
                    hx-get="{% url 'partial_search' %}"
                    hx-target="#load-more"
                    hx-vals='{"q": "{{ request.GET.q }}", "page": "{{ page.next_page_number }}"}'
                    hx-swap="outerHTML"
                >
                    Load More
                </button>
            </div>
        {% endif %}
    </div>
{% endif %}

  

hx-get: sends a GET request to partial-search/
hx-target: the <div> with id="load-more" which contains the Load More button.

For Load More button, we have 2 additional attributes we didn’t mention before:

hx-vals: add parameters to be submitted within the request. It must be defined as a valid JSON (e.g. '{"page": "{{ page.next_page_number }}", "q": "{{ request.GET.q }}"}').
- We send the next page variable and the current search value q as parameters to the GET request.
hx-swap: how the response will be swapped in relative to the target (hx-target).
- outerHTML replaces the entire target element (<div id="load-more">…</div>) with the response. There are many possible options for the value of this attribute.

YAY! 🎉 We did it! That’s how our partial search view with a Load More button comes about and it feels much better now!

Discussion

This is just the beginning… There is so much more but the htmx docs are a great place to start and discover what else is possible. What I can say is: almost anything you want to do is feasible and there are multiple ways to accomplish that.

Some ideas from here:

In our example, we are using a simple function based views but it can definitely work for class-based views. That’s a good challenge to start if you want to try it out.
Load More button is not necessary the best option for all list views. If you want to have the pagination and keep the dynamic functionality, it’s totally possible to do it. That’s something you can also try it out.

📢 Now, tell me… Are you already using HTMX? What are the most interesting use-cases for which you have used Django with HTMX?

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to _deploy_ those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

Deploying Django to Production

2023-02-23T00:00:00+00:00

This post is about providing some guidance on how to make your simple Django app production-ready and deploying to Fly.io. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

The first Django app I ever created was a simple Blog back in 2015, during a Django Girls event in Brazil. Ever since then, I’ve created and deployed many other Django applications but the deployment process was never so easy as it was with Fly.io! I’m super excited to share it with you, let’s go? ✨

What I can find here?

This post has two main sections:

Setting up: steps on how to make your local Django app ready for going to production on any hosting platform.
Deploying to Fly.io: deploying your production-ready Django app to Fly.io.

For reference, we'are using Python 3.10.9 and Django 4.1.6.

For this guide, I’ll be updating the Django Girls Tutorial, a server-rendered Blog, to exemplify how we can transform any Django app to be a production-ready application independent of the hosting provider.

Then, we’ll go through the deployment process to Fly.io. If you already have a production-ready app, that’s great! You might want to jump to the “Deploying to Fly.io” section.

With all that said, let’s start at the beginning…

Setting up ⚒️

We assume the initial setup is already done, you have Python installed and a virtual environment created and activated to manage our dependencies. We’ll be using venv for this project:

    # Unix/macOS
$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $

# Windows
$ python -m venv .venv
$ .venv\Scripts\activate
(.venv) $

  

From this point on, the commands won’t be displayed with (.venv) $ but we assume you have your Python virtual environment activated.

Database

Django uses SQLite by default and this is the easiest way to start once no other packages are required to support this database. However, we want a more scalable database so we decide to use PostgreSQL on production. According to the official documentation, PostgreSQL is the best-suited and supported database for Django and also the most used database by Django developers, as reported by the official annual Django Developers Survey 2021.

The great thing is that Fly.io provides a single-node/high availability PostgreSQL cluster out of the box for us to use. It’s also easy to set it up when configuring your deployment. We’ll go over how in the next steps.

For now, make sure to download and install PostgreSQL. You can go ahead and create your local database. For my app, the database is called blog.

The config updates required to change our database are explained in the next steps.

Environment Variables

First of all, we want to store the configuration separate from our code and load them at runtime. This allow us to keep one settings.py file and still have multiple environments (i.e. local/staging/production).

One popular options is the usage of the environs package. Another option is the python-decouple package, which was originally designed for Django, but it’s recommended to be used with dj-database-url to configure the database. For this guide we’ll use django-environ to configure our Django application, which allows us to use the 12factor approach.

Make sure your Python virtual environment is activated and let’s install the django-environ:

python -m pip install django-environ==0.9.0

In our settings.py file, we can define the casting and default values for specific variables, for example, setting DEBUG to False by default.

    # settings.py
from pathlib import Path
import environ  # <-- Updated!

env = environ.Env(  # <-- Updated!
    # set casting, default value
    DEBUG=(bool, False),
)

  

django-environ (and also the other mentioned packages) can take environment variables from the .env file. Go ahead and create this file in your root directory and also don’t forget to add it to your .gitignore in case you are using Git for version control (you should!). We don’t want those variables to be public, this will be used for the local environment and be set separately in our production environment.

Make sure you are taking the environment variables from your .env file:

    # settings.py
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent

# Take environment variables from .env file
environ.Env.read_env(BASE_DIR / '.env')  # <-- Updated!

  

We can now set the specific environment variables in the .env file:

    # .env
SECRET_KEY=3ohiu^m1su%906rf#mws)xt=1u#!xdj+l_ahdh0r#$(k_=e7lb
DEBUG=True

  

Check that there are no quotations around strings neither spaces around the =.

Coming back to our settings.py, we can then read SECRET_KEY and DEBUG from our environment variables:

    # settings.py
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = env('SECRET_KEY')  # <-- Updated!

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = env('DEBUG')  # <-- Updated!

  

The last environment variable to be set is the DATABASE_URL. env.db() will read from the DATABASE_URL variable.

    # settings.py
DATABASES = {
    # read os.environ['DATABASE_URL']
    'default': env.db()  # <-- Updated!
}

  

Here we can define our local database, adding it to the .env file:

    # .env
SECRET_KEY=3ohiu^m1su%906rf#mws)xt=1u#!xdj+l_ahdh0r#$(k_=e7lb
DEBUG=True
DATABASE_URL=postgres://postgres:postgres@localhost:5432/blog  # <-- Updated!

  

If you prefer to keep using SQLite on your local environment, that’s where you should specify:

    # .env
SECRET_KEY=3ohiu^m1su%906rf#mws)xt=1u#!xdj+l_ahdh0r#$(k_=e7lb
DEBUG=True
DATABASE_URL=sqlite:///db.sqlite  # <-- Updated!

  

⚠️ However, it’s highly recommended to use the same database for development and production to avoid inconsistent behavior between different environments.

Psycopg

To interact with our Postgres database, we’ll use the most popular PostgreSQL database adapter for Python, the psycopg package. With your virtual environment activated, go ahead and installed it:

python -m pip install psycopg2==2.9.5

⚠️ For production, it’s advised to use the source distribution instead of the binary package (psycopg2-binary).

Gunicorn

When starting a Django project (with startproject management command), we get a minimal WSGI configuration set up out of the box. However, this default webserver is not recommended for production. Gunicorn (Green Unicorn) is a Python WSGI HTTP Server for Unix and one of the easiest to start with. It can be installed using pip:

python -m pip install gunicorn==20.1.0

Static Files

Handling static files in production is a bit more complex than in development. One of the easiest and most popular ways to serve our static files in production is using the WhiteNoise package, which serves them directly from our WSGI Server (Gunicorn). Install it with:

python -m pip install whitenoise==6.3.0

A few changes to our settings.py are necessary.

Add the WhiteNoise to the MIDDLEWARE list right after the SecurityMiddleware.
Set the STATIC_ROOT to the directory where the collectstatic management command will collect the static files for deployment.
(Optional) Set STATICFILES_STORAGE to CompressedManifestStaticFilesStorage to have compression and caching support.

    # settings.py
MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',  # <-- Updated!
    ...
]

...
STATIC_ROOT = BASE_DIR / 'staticfiles'  # <-- Updated!

STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'  # <-- Updated!

  

It’s recommended to use WhiteNoise also in development to keep consistent behavior between development and production environments. The easiest way to do that is to add whitenoise.runserver_nostatic to our INSTALLED_APPS right before the built-in staticfiles app:

    # settings.py
INSTALLED_APPS = [
    ...
    'whitenoise.runserver_nostatic',  # <-- Updated!
    'django.contrib.staticfiles',
    ...
]

  

`ALLOWED_HOSTS` and `CSRF_TRUSTED_ORIGINS`

As a security measure, we should set in ALLOWED_HOSTS, a list of host/domain names that our Django website can serve. For development we might include localhost and 127.0.0.1 and for our production we can start with .fly.dev (or the provider’s subdomain you chose) and update for the dedicated URL once your app is deployed to the hosting platform.

CSRF_TRUSTED_ORIGINS should also be defined with a list of origins to perform unsafe requests (e.g. POST). We can set the subdomain https://*.fly.dev (or the provider’s subdomain you chose) until our deployment is done and we have the proper domain for our website.

    # settings.py
ALLOWED_HOSTS = ['localhost', '127.0.0.1', '.fly.dev']  # <-- Updated!

CSRF_TRUSTED_ORIGINS = ['https://*.fly.dev']  # <-- Updated!

Installed packages

Make sure you have all necessary installed packages tracked and listed in your requirements.txt by running:

pip freeze > requirements.txt

This command generates the requirements.txt file if it doesn’t exist.

    # requirements.txt
asgiref==3.6.0
Django==4.1.6
django-environ==0.9.0
gunicorn==20.1.0
psycopg2==2.9.5
sqlparse==0.4.3
whitenoise==6.3.0

  

With our Django application prepped and ready for production hosting, we’ll take the next step and deploy our app to Fly.io!

Deploying to Fly.io 🚀

flyctl is the command-line utility provided by Fly.io.

If not installed yet, follow these instructions, sign up and log in to Fly.io.

Launching our App

Fly.io allows us to deploy our Django app as long as it’s packaged in a Docker image. However, we don’t need to define our Dockerfile manually. Fly.io detects our Django app and automatically generates all the necessary files for our deployment. Those are:

Dockerfile contain commands to build our image.
.dockerignore list of files or directories Docker will ignore during the build process.
fly.toml configuration for deployment on Fly.io.

All of those files are templates for a simple Django app and can be modified according to your needs.

Before deploying our app, first we need to configure and launch our app to Fly.io by using the flyctl command fly launch. During the process, we will:

Choose an app name: this will be your dedicated fly.dev subdomain.
Select the organization: you can create a new organization or deploy to your personal account (connect to your Fly account, visible only to you).
Choose the region for deployment: Fly.io initially suggests the closest to you, you can choose another region if you prefer.
Set up a Postgres database cluster: flyctl offers a single node “Development” config that is designed so we can turn it into a high-availability cluster by adding a second instance in the same region. Fly Postgres is a regular app you deploy on Fly.io, not a managed database.

This is what it looks like when we run fly launch:

fly launch

Creating app in ../flyio/katias-blog
Scanning source code
Detected a Django app
? Choose an app name (leave blank to generate one): katias-blog
? Select Organization: Kátia Nakamura (personal)
Some regions require a paid plan (bom, fra, maa).
See https://fly.io/plans to set up a plan.

? Choose a region for deployment: Frankfurt, Germany (fra)
App will use ‘fra’ region as primary

Created app ‘katias-blog‘ in organization ‘personal‘
Admin URL: https://fly.io/apps/katias-blog
Hostname: katias-blog.fly.dev
Set secrets on katias-blog: SECRET_KEY  <-- # SECRET_KEY is set here!
? Would you like to set up a Postgresql database now? Yes
? Select configuration: Development - Single node, 1x shared CPU, 256MB RAM, 1GB disk
Creating postgres cluster in organization personal
Creating app...
Setting secrets on app katias-blog-db...
Provisioning 1 of 1 machines with image flyio/postgres:14.6
Waiting for machine to start...
Machine 32874445c04218 is created
==> Monitoring health checks
  Waiting for 32874445c04218 to become healthy (started, 3/3)

Postgres cluster katias-blog-db created
  Username:    postgres
  Password:    <your-internal-postgres-password>
  Hostname:    katias-blog-db.internal
  Proxy port:  5432
  Postgres port:  5433
  Connection string: postgres://postgres:<your-internal-postgres-password>@katias-blog-db.internal:5432

Save your credentials in a secure place -- you won't be able to see them again!

Connect to postgres
Any app within the Kátia Nakamura organization can connect to this Postgres using the above connection string

Now that you've set up Postgres, here's what you need to understand: https://fly.io/docs/postgres/getting-started/what-you-should-know/
Checking for existing attachments
Registering attachment
Creating database
Creating user

Postgres cluster katias-blog-db is now attached to katias-blog
The following secret was added to katias-blog:  <-- # DATABASE_URL is set here!
  DATABASE_URL=postgres://katias_blog:<your-postgres-password>@top2.nearest.of.katias-blog-db.internal:5432/katias_blog?sslmode=disable
Postgres cluster katias-blog-db is now attached to katias-blog
? Would you like to set up an Upstash Redis database now? No
Creating database migrations
Wrote config file fly.toml

[INFO] Python 3.10.12 was detected. ‘python:3.10-slim-buster’ image will be set in the Dockerfile.

Validating .../flyio/katias-blog/fly.toml
Platform: machines
✓ Configuration is valid

‘STATIC_ROOT’ setting was detected in website/settings.py’!
Static files will be collected during build time by running ‘python manage.py collectstatic’ on Dockerfile.

Your Django app is ready to deploy!

For detailed documentation, see https://fly.dev/docs/django/

During the process, the SECRET_KEY and DATABASE_URL will be automatically set to be used on your production deployment. Those are the only ones we need at the moment but if you have any other secrets, check here how to set them. You can also list all your application secret names:

fly secrets list

NAME            DIGEST                  CREATED AT
DATABASE_URL    cc999c17fa021988        2023-02-07T19:48:55Z
SECRET_KEY      e0a6dbbd078004f7        2023-02-07T19:47:33Z

fly launch sets up a running app, creating the necessary files: Dockerfile, .dockerignore and fly.toml.

During the fly launch, your wsgi file should have been detected automatically. If not, don’t forget to replace demo.wsgi in your Dockerfile with your Django project’s name:

    # Dockefile
...
# replace demo.wsgi with <project_name>.wsgi
CMD ["gunicorn", "--bind", ":8000", "--workers", "2", "website.wsgi"]  # <-- Updated!

  

For production, it’s advised to use psycopg2 package (instead of the psycopg2-binary) built from source but it requires external packages to work properly: libpq-dev is a very light package and provides all requirements for building psycopg2 and gcc is a compiler used to install psycopg2. Let’s install them:

    # Dockerfile
...
WORKDIR /code

# install psycopg2 dependencies
RUN apt-get update && apt-get install -y \
    libpq-dev \
    gcc \
    && rm -rf /var/lib/apt/lists/*  # <-- Updated!

COPY requirements.txt /tmp/requirements.txt
...

  

For security reasons, we’ll add .env to our .dockerignore file - so Docker doesn’t include our secrets during the build process.

# .dockerignore
fly.toml
.git/
*.sqlite3
.env  # <-- Updated!

This means the environment variables (i.e. SECRET_KEY) stored in the .env file won’t be available at build time, neither the secrets automatically set by Fly.io on your application during the fly launch process.

We have two options here:

Set a default SECRET_KEY using the get_random_secret_key function provided by Django that will be used at build time. At runtime, we’ll use the SECRET_KEY set by Fly.io, i.e. the default value only applies to the build process.

    # settings.py
from django.core.management.utils import get_random_secret_key
...
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = env.str('SECRET_KEY', default=get_random_secret_key())  # <-- Updated!

  

Another option is to set a “non-secret” dummy SECRET_KEY on Dockerfile only for building purposes:

    # Dockerfile

# Set SECRET_KEY for building purposes
ENV SECRET_KEY "non-secret-key-for-building-purposes"  # <-- Updated!
RUN python manage.py collectstatic --noinput

This keeps our environment variables safe and makes sure they will be set for the different environments.

In case your app contains static files such as images, CSS or Javascript files, we need to collect all the static files into a single location and make them accessible to be served in production. This process needs to happen at build time (so they are persisted when building our image) by running collectstatic command on Dockerfile.

If you noticed a similar output:

...
‘STATIC_ROOT’ setting was detected in website/settings.py’!
Static files will be collected during build time by running ‘python manage.py collectstatic’ on Dockerfile.

your Dockerfile was already updated to run the collectstatic command.

On the other hand, if STATIC_ROOT wasn’t detected automatically in your settings.py file and your app contains static files to be served, make sure to collect them during the build time by adding this step to the Dockerfile:

    # Dockerfile
...
RUN python manage.py collectstatic --noinput

  

Now that we have set our app name, we can update our settings.py with the dedicated subdomain we chose (or that was generated for us):

    # settings.py
ALLOWED_HOSTS = ['localhost', '127.0.0.1', 'katias-blog.fly.dev']  # <-- Updated!

CSRF_TRUSTED_ORIGINS = ['https://katias-blog.fly.dev']  # <-- Updated!

Finally, in the [[statics]] section on fly.toml file, we define the guest_path, which is the path inside our container where the files will be served directly to the users, bypassing our web server. In the settings.py we defined the STATIC_ROOT:

    # settings.py
STATIC_ROOT = BASE_DIR / 'staticfiles'  # <-- Updated!

The Dockerfile generated by fly launch defines our WORKDIR as /code. That’s where our static files will be collected: /code/staticfiles. Let’s go ahead and update our fly.toml to serve those files directly:

    # fly.toml
...
[[statics]]
  guest_path = "/code/staticfiles"  # <-- Updated!
  url_prefix = "/static"

  

Deploying our App

Great! All ready and it’s finally time to deploy our app:

fly deploy

...
  Finished deploying

Visit your newly deployed app at https://katias-blog.fly.dev/

Our app is now up and running! ⚙️ Try:

fly open

💾 By the way, you can find the example code used in this guide here.

YAY! 🎉 We just deployed our Django app to production! How great is that?

What’s next?

Make sure to check the official deployment checklist provided by Django docs with more details on specific settings we might not have covered in this guide. You can also check the Fly.io Docs - we’re currently working on the Django on Fly.io docs!

If you have any question or comments, reach out on the Fly.io Community. That’s a great place to share knowledge, help and get help!

📢 Now, tell me… What are your go-to packages 📦 to make your Django app production-ready?

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to _deploy_ those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

Standout features in Django 4.2

2023-02-02T00:00:00+00:00

This post is about the new Django 4.2 release. It’s got some neat things in it and Mariusz Felisiak shares his favorite highlights. Django on Fly.io is pretty sweet! Check it out: you can be up and running on Fly.io in just minutes.

After 8 months of work by over 200! contributors 💗, the first alpha version of Django 4.2 is out! This is a long-term support release (LTS) with extended support until April 2026, so 3 more years.

The final release should be issued in early April 2023, so now is the best time to take a peek 👀 at a “farrago” of new features shipped to this magnificent release 📦. I’d like to take a moment and share my personal favorites of Django 4.2 goodies.

`psycopg` version 3 support

Django 4.2 supports psycopg version 3.1.8+ which is the new implementation of the most popular and the richest PostgreSQL adapter for Python. The best part is that there is no need to change the ENGINE as the built-in django.db.backends.postgresql backend supports both libraries, psycopg2 and psycopg. It’s enough to install psycopg in your environment:

    python -m pip install "psycopg>=3.1.8"

  

psycopg prefers server-side parameter binding which improves performance and memory usage. It’s disabled by default in Django as it causes hiccups in some cases e.g. parameters passed to expressions in SELECT and GROUP BY clauses are not recognized as the same values which can cause grouping errors. Even though support for server-side binding cursors is still experimental it’s worth trying. If you want to use it, set the server_side_binding option in your DATABASES configuration:

    DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        # ...
        "OPTIONS": {
            "server_side_binding": True,
        },
    },
}

  

psycopg also provides asynchronous connections and cursors which should take Django async support to the next level in the future 🔮.

Comments on columns and tables

Database comments on columns and tables are really helpful for keeping your database schema maintainable. It also helps create a bridge 🤝 between developers and people with direct access to the database like database administrators or data scientists.

11 years after creating the ticket 🗝️, Django 4.2 added support for table and column comments via the new Field.db_comment and Meta.db_table_comment options (for all database backends included with Django except SQLite). The migrations framework will propagate comments to your tables metadata. For example:

    from django.db import models

class Product(models.Model):
    name = models.CharField(max_length=511, db_comment="Product name")
    deleted = models.BooleanField(
        default=False,
        db_comment=(
            "Soft delete. When value is `True`, product is not visible "
            "for users. This is useful for storing data about products "
            "that are no longer for sale."
        ),
    )

    class Meta:
        db_table_comment = "Available products"

  

In general, table and column comments are not for application users, however storing them in the model description allows you to create a single place where our database schema is managed. As such, it covers another place where you can use the Django ORM instead of writing raw SQL statements.

Lookups on field instances

Registering lookups on Field instances can be really helpful, especially if you are familiar with the Lookup API:

A lookup is a query expression with a left-hand side, lhs; a right-hand side, rhs; and a lookup_name that is used to produce a boolean comparison between

Django < 4.2 only allowed registering lookups on Field classes, e.g. __abs lookup for IntegerFied. Django 4.2 takes the use of specialized lookup to the next and extremely flexible level when you can have a different set of lookup for each field in each model. This allows for creating reusable hooks and can reduce the number of annotations used when filtering a queryset. Let’s assume you have a shop system, with a Product model and you want to filter out expired products.

    from django.db import models

class Product(models.Model):
    name = models.CharField(max_length=511)
    best_before = models.DateTimeField()
    ...

  

You can do this by comparing the best_before field with the current date:

    >>> from django.utils import timezone
>>> Product.objects.filter(best_before__lt=timezone.now())
<QuerySet [<Product: Milk>]>

  

In Django 4.2, you can now create a lookup IsOverdue:

    from django.db import models
from django.db.models.functions import Now

class IsOverdue(models.Lookup):
    lookup_name = "is_overdue"
    prepare_rhs = False

    def as_sql(self, compiler, connection):
        if not isinstance(self.rhs, bool):
            raise ValueError(
                "The QuerySet value for an is_overdue lookup "
                "must be True or False."
            )
        sql, params = self.process_lhs(compiler, connection)
        now_sql, now_params = compiler.compile(Now())
        if self.rhs:
            return f"{sql} < {now_sql}", (*params, *now_params)
        else:
            return f"{sql} >= {now_sql}", (*params, *now_params)

  

    Product._meta.get_field("best_before").register_lookup(IsOverdue)

  

and use it as a handy shortcut:

    >>> Product.objects.filter(best_before__is_overdue=True)
<QuerySet [<Product: Milk>]>

Closing Thoughts

Here we dug a little deeper on just 3 of my favorite improvements in the Django 4.2 release. These features all happen to be related to database support and I consider them solid quality of life improvements. Of course, there are a lot of other fixes and improvements in this release and I strongly encourage you to check the release notes. A couple other features worth mentioning are the accessibility improvements in the admin site and forms, and the constantly improving asynchronous support.

All in all, Django 4.2 is another solid release of one of the most popular web frameworks. What enhancements or fixes are you most looking forward to?

Give it a spin and share!

Django really flies on Fly.io

You already know Django makes it easier to build better apps. Well now Fly.io makes it easier to _deploy_ those apps and move them closer to your users making it faster for them too!

Deploy a Django app today! →

Django Beats

ASGI deployment options for Django

ASGI application

Daphne

Hypercorn

Uvicorn

Granian

Closing thoughts

Django really flies on Fly.io

Shedding light on Django versioning

Release cycle

Stability

Python support

Closing thoughts

Django really flies on Fly.io

Django Views as Serverless Functions on Fly Machines

Fly Machines processes

Fly Machines services

fly-replay 🔁

Django Views as Serverless Functions ✨

views.py

urls.py

settings.py

Fly Machines

FlyReplayMiddleware

Try it out 🚀

Final Thoughts

Django really flies on Fly.io

Staging environments on the Fly with GitHub actions

Set up

Basic flow

Use Postgres cluster

Additional release steps

Serverless Functions in Django With Fly Machines

Serverless Functions 🚀

Serverless Functions in Django

Fly Machines 🤖

Fly Machines API to the Rescue

Django Commands as Serverless Functions ✨

Fly Machine via Machines API

Machines App

Sentiment App

Final Thoughts: Limitations and Flaws

Django really flies on Fly.io

New goodies in Django 5.0

Generated fields

Database-computed default values

Form field group rendering

Closing thoughts

Smooth Database Changes in Blue-Green Deployments

Migrations framework

Workflow

Separate database and project state

Full example

Closing Thoughts

Celery Async Tasks on Fly Machines

Enter Celery…

Backend and Brokers

Django ❤️ Celery

Our Application 📰

Setting Up 🛠

Running Redis Locally

Configuration

Running Celery Worker Server Locally

Storing the Results

Deploying to Fly.io 🚀

Fly Apps

Fly Machines

Scaling the Number of Machines

Django really flies on Fly.io

What’s Next?

Organizing Database Queries: Managers vs. QuerySets

Problem

Managers and QuerySets

Full example

Soft-delete with a custom Manager

Sharing and chaining more annotations, aggregations, and filters

Closing Thoughts

QuerySets are Lazy

What are Lazy QuerySets? 💭

Fly Machines `processes`

Fly Machines `services`

`fly-replay` 🔁

Displaying Queries with `django-debug-toolbar`

`fly deploy`

`django-htmx`

`ALLOWED_HOSTS` and `CSRF_TRUSTED_ORIGINS`

`psycopg` version 3 support