Custom Domains and SSL Certificates

For a quick custom domain how-to, see Use a custom domain.

An application’s brand is often encapsulated in its domain name, and that in turn is wrapped with value. So being able to configure secure custom domains is essential.

Fly.io offers a simple command-line process for manual configuration of custom domains and a GraphQL API for people integrating Fly.io custom domains into their automated workflows. Here, we’ll be looking at both - and answering the question, which one should you use?

Teaching your app about custom domains

Your application code needs to know how to accept custom domains and adjust the responses accordingly. If you’re hosting content on behalf of your users, this typically means mapping the incoming hostname to a particular ID in your application database.

When users make requests, their browser sends a Host header you can use to alter the behavior of your application. When you run your app server on Fly.io directly, just get the contents of the Host header to identify a request.

If you’re running your application on another provider, you will need to create a proxy application (like NGINX) to route traffic through Fly.io. Your application can then use the X-Forwarded-Host header to determine how to handle requests.

Creating a custom domain on Fly.io manually

There’s a question to ask and answer. Do you want to start accepting traffic immediately on your custom domain or do you want to have your domain ready with certificates when you set it to start accepting traffic, for example when you want to cut over from another platform to Fly.io.

Accepting traffic immediately for the custom domain

In this scenario, we want the custom domain to point to the nginxproxy server which will allow unencrypted IPv4 and IPv6 connections. Again, there are two ways to do this. Using DNS CNAME capability or setting the A and AAAA records.

Option I: CNAME records

CNAME records in DNS act like a pointer. If we add a CNAME record to our custom domain that points to our proxy name nginxproxy.fly.dev then requests for the custom domain’s IP address would return the proxy’s address and clients would then lookup the IP addresses for the proxy.

It’s the quickest way to get set up, but there are catches. First, it is ever so slightly slower with that second look up. Second, it limits what you can do with the domain, especially if it’s an “Apex domain” - CNAME records are, according to DNS standards, meant to be the only record in a host’s DNS records and so you can’t add MX and other essential records to the DNS entry. If you aren’t setting up an Apex domain, the CNAME is the quickest way to get going.

Option II: A and AAAA records

We can skip all CNAME concerns by setting A and AAAA records on your custom domain. Run fly ips list to see your app’s addresses:

fly ips list

  TYPE   ADDRESS                                CREATED AT
  v4     77.83.143.105                          2020-03-02T14:59:13Z
  v6     2a09:8280:1:659f:6cb7:4925:6bfd:90a3   2020-03-02T14:59:13Z

Create an A record pointing to your IPv4 address, and an AAAA record pointing to your IPv6 address. You’re then free to make this an Apex domain as needed.

Adding the certificate

Once these settings are in place, you can add the custom domain to the application’s certificates. If we are configuring example.com, then we would simply run:

fly certs create example.com

If you need a wildcard domain, remember to place quotes around the hostname to avoid shell expansion:

fly certs create "*.example.com"

This will kick off the process of validating your domain and generating certificates. You can check on the progress if you run:

fly certs show example.com

  Hostname                    = example.com
  Configured                  = true
  Issued                      = rsa,ecdsa
  Certificate Authority       = lets_encrypt
  DNS Provider                = dnsimple
  DNS Validation Instructions = CNAME _acme-challenge.example.com => example.com.o055.flydns.net.
  DNS Validation Hostname     = _acme-challenge.example.com
  DNS Validation Target       = example.com.o055.flydns.net
  Source                      = fly
  Created At                  = 1m9s ago
  Status                      = Ready

Configured should be true and Status will show Ready when the certificates are available. The Issued field shows which types of certificates are available: RSA and/or ECDSA. Once they are issued, you’ll be ready to run with your custom domain.

Configuring certificates before accepting traffic

This process allows certificates to be issued before the domain is live and accepting traffic. Using a particular CNAME entry in the DNS allows Fly.io to validate the domain before issuing a certificate.

The process starts with adding the certificate to the application like so:

fly certs create example.com

Again, if you are creating a wildcard domain, remember to place quotes around the hostname to avoid shell expansion:

fly certs create "*.example.com"

Now we have created the certificate, we need to ask for the CNAME entry that has to be set up. For this, we run fly certs show:

fly certs show example.com

  Hostname                    = example.com
  Configured                  = false
  Issued                      =
  Certificate Authority       =
  DNS Provider                = dnsimple
  DNS Validation Instructions = CNAME _acme-challenge.example.com => example.com.o055.flydns.net.
  DNS Validation Hostname     = _acme-challenge.example.com
  DNS Validation Target       = example.com.o055.flydns.net
  Source                      = fly
  Created At                  = 0m9s ago
  Status                      =

The specific part to focus on here are the DNS Validation fields:

DNS Validation Instructions = CNAME _acme-challenge.example.com => example.com.o055.flydns.net.
  DNS Validation Hostname     = _acme-challenge.example.com
  DNS Validation Target       = example.com.o055.flydns.net

Basically, the Validation Hostname, when looked up, should send requests to the Validation Target, a Fly.io-generated validation service. To do this, add the contents of the Validation Instructions to your DNS records; that is create a CNAME record which points the _acme-challenge subdomain to the Validation Target.

Once you have done that, wait as the validation and certificate issuing happens (check in with fly certs check). When complete, you’ll be able to turn on the traffic whenever you are ready. You’ll be able to do that either by setting the CNAME or by setting the A and AAAA records as described previously.

Other commands

Finally, we should point out the other fly certs commands.

• fly certs list - Lists the hostnames which have been added to an application and for which certificates may have been obtained.
• fly certs check hostname - Triggers a check on the domain validation and DNS configuration for the given host and return results in the same format as fly certs show.
• fly certs delete hostname - Removes the hostname from the application, dropping the certificates in the process.

Automating the certificate process

To illustrate how to automate the certificates API, we are going to show the flyctl command and the equivalent GraphQL request, wrapped in a compact easy-to-read Node application from our fly-examples/hostnamesapi repository.

GraphQL API Notes

Endpoints: The Endpoint for the Fly.io GraphQL API is https://api.fly.io/graphql.

Authentication: All queries require an API token which be obtained by signing into the Fly.io web dashboard, selecting Account ➡︎ Settings ➡︎ Access Tokens ➡︎ Create Access Token. Create a new token and carefully note the value; we suggest placing it in an environment variable such as FLY_API_TOKEN so it can be passed to applications. When used with the API, the token should be passed in an Authorization header with the value Bearer <token value>.

IDs and Names: Applications can be referred to by name or by ID. Currently the Application ID and Name are interchangeable. This will change in a future semantically significant version. To see how to query for ID and Name, see getappbyid.js and getappbyname.js in the example repository.

Listing all the hosts of an application

With flyctl: fly certs list

With GraphQL: The example is in the repository as getcerts.js. This request takes the application name as a parameter.

query($appName: String!) {
  app(name: $appName) {
    certificates {
      nodes {
        createdAt
        hostname
        clientStatus
      }
    }
  }
}

Example output:

{
  "app": {
    "certificates": {
      "nodes": [
        {
          "createdAt": "2020-03-04T14:17:14Z",
          "hostname": "example.com",
          "clientStatus": "Ready"
        },
        {
          "createdAt": "2020-03-05T15:28:41Z",
          "hostname": "exemplum.com",
          "clientStatus": "Ready"
        }
          ]
      }
  }
}

This lists every host associated with the application. Each host may have up to two (RSA/ECDSA) certificates associated with it. See “Reading a certificate from an application” to see how to query the certificates associated with the hostnames.

Creating a certificate for an application

With flyctl: fly certs create <hostname>

With GraphQL: The example is addcert.js. This request takes the application id and hostname as parameters.

mutation($appId: ID!, $hostname: String!) {
    addCertificate(appId: $appId, hostname: $hostname) {
        certificate {
            configured
            acmeDnsConfigured
            acmeAlpnConfigured
            certificateAuthority
            certificateRequestedAt
            dnsProvider
            dnsValidationInstructions
            dnsValidationHostname
            dnsValidationTarget
            hostname
            id
            source
        }
    }
}

Example Output:

{
  "addCertificate": {
    "certificate": {
      "configured": true,
      "acmeDnsConfigured": true,
      "acmeAlpnConfigured": true,
      "certificateAuthority": "lets_encrypt",
      "certificateRequestedAt": "2020-03-06T12:26:36Z",
      "dnsProvider": "enom",
      "dnsValidationInstructions": "CNAME _acme-challenge.codepope.wtf => example.com.o055.flydns.net.",
      "dnsValidationHostname": "_acme-challenge.example.com",
      "dnsValidationTarget": "example.com.o055.flydns.net",
      "hostname": "example.com",
      "id": "LO7FgYIy0sBZC8yuGNFKQH4QCq4ujMfJZumJCVNiQxhMq",
      "source": "fly"
    }
  }
}

The returned data here includes all the values needed to configure DNS records for pre-traffic validation.

Reading a certificate from an application

With flyctl: fly certs show hostname

With GraphQL: The example is getcert.js. This request takes the application name and hostname as parameters.

query($appName: String!, $hostname: String!) {
  app(name: $appName) {
    certificate(hostname: $hostname) {
      configured
      acmeDnsConfigured
      acmeAlpnConfigured
      certificateAuthority
      createdAt
      dnsProvider
      dnsValidationInstructions
      dnsValidationHostname
      dnsValidationTarget
      hostname
      id
      source
      clientStatus
      issued {
        nodes {
          type
          expiresAt
        }
      }
    }
  }
}

Example Output:

{
  "app": {
    "certificate": {
      "configured": true,
      "acmeDnsConfigured": true,
      "acmeAlpnConfigured": true,
      "certificateAuthority": "lets_encrypt",
      "createdAt": "2020-03-04T17:17:39Z",
      "dnsProvider": "enom",
      "dnsValidationInstructions": "CNAME _acme-challenge.example.com => example.com.o055.flydns.net.",
      "dnsValidationHostname": "_acme-challenge.example.com",
      "dnsValidationTarget": "example.com.o055.flydns.net",
      "hostname": "example.com",
      "id": "4n8ikGIzjsm0s5VclwTDaSODtveu2xCZkHkKCaRilafGk",
      "source": "fly",
      "clientStatus": "Ready",
      "issued": {
        "nodes": [
          {
            "type": "ecdsa",
            "expiresAt": "2020-06-02T16:17:51Z"
          },
          {
            "type": "rsa",
            "expiresAt": "2020-06-02T16:17:45Z"
          }
        ]
      }
    }
  }
}

Most of the output duplicates the details from adding a certificate, including the DNS validation settings. The difference here is the issued section which contains an array of nodes, each of which is the type and expiry date of certificates that have been issued against this host name. Here we see that ECSDA and RSA certificates have been issued.

Checking a certificate

With flyctl: fly certs check hostname

With GraphQL: The example is checkcert.js. This request takes the application name and hostname as parameters. It is essentially the same as reading the certificate, but the presence of a request for the certificate’s check value will start a validation process. The output is similar too.

query($appName: String!, $hostname: String!) {
  app(name: $appName) {
    certificate(hostname: $hostname) {
        check
        configured
        acmeDnsConfigured
        acmeAlpnConfigured
        certificateAuthority
        createdAt
        dnsProvider
        dnsValidationInstructions
        dnsValidationHostname
        dnsValidationTarget
        hostname
        id
        source
        clientStatus
        issued {
          nodes {
              type
              expiresAt
          }
        }
    }
  }
}

Deleting a certificate

With flyctl: fly certs delete hostname

With GraphQL: The example is deletecert.js. This request takes the application name and hostname as parameters and will remove the hostname from the application.

   mutation($appId: ID!, $hostname: String!) {
        deleteCertificate(appId: $appId, hostname: $hostname) {
            app {
                name
            }
            certificate {
                hostname
                id
            }
        }
    }

The GraphQL mutation returns the app name and the hostname and certificate id that was removed.

Example Output:

{
  "deleteCertificate": {
    "app": {
      "name": "nginxproxy"
    },
    "certificate": {
      "hostname": "example.com",
      "id": "6AZc4AS6ysZgHPqFg7TGzIyofewuZnToxu6lUbBi8DHGQ"
    }
  }
}

Troubleshooting

Certificate creation or validation seems to hang, stall or fail

Let’s Encrypt™ is a free, automated, and open certificate authority that Fly.io uses to issue TLS certificates for custom domains. However, Let’s Encrypt™ imposes certain rate limits to ensure fair usage. If you encounter issues when creating or validating a certificate for a custom domain on Fly.io, it’s possible that you’ve hit these rate limits.

The following rate limits from Let’s Encrypt™ apply:

• Certificates per Registered Domain: 50 per week
• Duplicate Certificate limit: 5 per week
• Failed Validation limit: 5 failures per hostname, per hour

Note that certificate renewals don’t count against your Certificates per Registered Domain limit.

If you encounter issues when adding or validating a certificate for a custom domain on Fly.io, you can use the following methods to troubleshoot:

• Use Let’s Debug: A diagnostic tool/website to help figure out why you might not be able to issue a certificate for Let’s Encrypt™. Using a set of tests, it can identify a variety of issues, including: problems with basic DNS setup, problems with nameservers, rate limiting, networking issues, CA policy issues and common website misconfigurations.
• Wait and Retry: If you’ve hit a rate limit, you’ll need to wait until the rate limit window passes before you can successfully create or validate a certificate again. We don’t have a way to reset it.

Remember, the best way to avoid hitting rate limits is to use staging environments and domains for testing and development, and to carefully plan your certificate issuance to stay within the limits. Avoid failed validation by ensuring that your DNS records are correctly configured, with no conflicting records.

If you’re building a platform on top of Fly.io, and you expect that your users will frequently delete and then recreate the same resources within a short window, consider implementing “soft delete” logic into your platform that retains the Fly.io resources for a period of time, negating the need to recreate certs frequently.

I use Cloudflare, and there seems to be a problem issuing or validating my Fly.io TLS certificate

If you’re using Cloudflare, you might be using their Universal SSL feature which inserts a TXT record of _acme_challenge.<YOUR_DOMAIN> for your domain. This can interfere with our certificate validation/challenge and you should disable this feature.

You can then verify that the change has propagated and the TXT record is no longer present by running dig txt _acme-challenge.<YOUR_DOMAIN> +short.

Wrapping up

You have everything you need to either hand assign a custom domain to your Fly.io application or to create your own automated multi-domain proxy.