How We Deliver Global SSL with Let's Encrypt

By Kellen 

Fly is proud to sponsor Let's Encrypt. We've been hard at work making Let's Encrypt TLS certificates as simple and safe as possible for developers and creators of all kinds. Within this article we'll explore how Fly applies Let's Encrypt certificates to servers around the world, highlight the benefits of Fly + Let's Encrypt, and ponder what the future has in store.


The Mission...

Phew... Making web-scale security features is no small feat. Luckily, there are supportive organizations and standardizations that help guide safe and effective implementations.

One such organization is Let's Encrypt, a non-profit maintained by the Internet Security Research Group (ISRG), a consortium of industry professionals based out of California. The functionality of Let's Encrypt can be broken into two flavours: Domain Validation, and Certificate Issuance and Revocation.

One such standardization is the Automatic Certificate Management Environment (ACME). ACME is a working document that aims to normalize how security certificates are orchestrated on the Internet. The initiative was created by Let's Encrypt and then standardized by the Inter Engineering Task Force (IETF) -- an international group whose goal is to make the Internet work better.

Both Let's Encrypt and ACME are central to how Fly serves HTTPS to the masses, and the masses-masses, from a global network.

Let's Encrypt... Flyly

When you create a project within Fly, your first step is to add a hostname. You provide the hostname, we provide a preview domain, and then you create a DNS record with the preview domain. After that, a Let's Encrypt HTTPS certificate is issued to that domain.

The ACME standard in motion here is Key Authorization. In order for a certificate to be generated control of the domain to sign must be proven. We've experimented with two of the available methods: DNS and HTTP. After running into an additional step with DNS challenges which required users to do an addiitional back-and-forth, we settled on HTTP verification.

Within HTTP verification the client proves itself by provisioning resources. The server challenges the client, demanding that it provision a specific value for a specific key. The authentication flow looks like this...

The server makes a challenge against the client; it's asking: "If you'd like a signed certificate, I need to make sure you control this domain. I'd like to see you provide an HTTP response that includes this token".

GET /.well-known/acme-challenge/LoqXcYV8q5ONbJQxbmVrcifhHcBvl3MUfjoTljzftw

The client then responds by appending the token together with the client's account key; it follows this formula:

key-authz = token || '.' || base64url(JWK_Thumbprint(accountKey))

In actuality, it might look something like this; it's saying: "Yes, I can respond with that token. Here's my token and here's my account key!"

HTTP/1.1 200 OK

On an individual basis, this wouldn't be too complex. Add a global network that contains many points-of-presence and things gets much more sophisticated. Each and every proxy that receives an HTTPS request needs to respond with the correct token to satisfy the challenge.

To accomplish this, the moment that a tokenized response exists we update a centralized instance of Redis. The certificates are stored within Redis as encrypted blobs. The blobs are replicated to each proxy. Each proxy authenticates against AWS' KMS and - if accepted - can decrypt the blob.


Within the proxy, active certificates are only stored within memory. The end result is that each proxy can draw the correct token from Redis. Each one can satisfy the challenge and prove control in a secure way. But, there's more to it!

We need to make sure we cover all of the cases. If an HTTPS request is made and there is no certificate, we begin the challenge attempt, distribute the key, and ensure that the HTTPS request travels along with it. Running health-checks ensures that we have contingency in this case. We don't want to spend extra milliseconds waiting for the response; we'd like to be prepared in the event that there is no certificate, so the experience is seamless.

Another thing to consider is that when providing HTTPS to a wide array of individuals, you will need to support an equally wide array of platforms. We provide both modern ECDSA (in beta) and older RSA certificates; that way, no matter which browser our customers or customers' customers visit, they'll have secure HTTPS.

Friends with Benefits

Encrypt-y Goodness

The key benefit is HTTPS for everyone! Let's Encrypt is making the Internet safer through HTTPS ubiquity. Fly uses Let's Encrypt as the underlying mechanism for performance optimized TLS; it's simple and can be applied whether you're paying or not. What's so good about Let's Encrypt, anyway?

There are two key aspects that make them special:

These factors in tandem allow services like Fly to provide TLS without becoming an intermediate authority; intermediate authorities are peculiar given the linear establishment of trust between server and client.

We can't sign for domains we can't verify; through Let's Encrypt and ACME, you are ensured that the certificate you receive is in good standing and vouches for domain control and the presence of a TLS certificate. Given how important it is that web entities are signed for by an objective and detached party, the democratized vision for certificate generation is a welcome one. We are proud to support it.

We've written in-depth about HTTPS, if you'd like to freshen up on the present and future of HTTPS.


When hosting Let's Encrypt yourself, you're in charge of configuring the renewal of your certificates. They expire after 90 days; good form would have you renewing every 60 days. If you have many domains to verify, or many servers to verify upon, that becomes a cumbersome task of cronjobbery and scripting. With Fly, we take care of the renewal process. You set it up, then jump back into developing things.

We have a handy guide on how you can set-up it yourself. It's, err-- a bit more tricky.

Global Deliverance

By using Redis to distribute verification tokens to each Fly edge-proxy, TLS is terminated around the globe. No matter where your users are located they won't be bogged down by lengthy TLS negotiations. When you factor in edge-caching and HTTP/2 your applications will receive an ample speed-boost.

HTTPS is simple to apply; one just needs to click a button and it's activated. Caching brings assets closer to your visitors, reducing distance and therefore latency; activate the Middleware and then configure it how you wish. HTTP/2 requires some strategic shifting to your development practices but results significant performance improvements in its own right.

If you'd like to know how to enjoy the benefits of HTTP/2 -- you guessed it, guide for that.


The Fly API is a powerful tool. You are but one POST away from plugging a Let's Encrypt TLS-signed hostname into the Fly network:

curl -H "Authorization: Bearer 9f5e850b7d5fac5566aa35e5a3836df03eac4b3c79115e1f19d43d979ddada48" \

-H "Content-Type: application/json" \

-X POST -d '{"data": { "attributes": { "hostname": "" } } }' \`

... Things churn and bubble...


{"data":{"id" :"fly-project","type":"hostnames","attributes":{"hostname":"","preview_hostname":""}}}

Bam! You - or your users for whom you have generated a custom hostname - take the preview_hostname and create a DNS record. That's it, that's all. Let's Encrypt woven into a suped-up Global Network, but one call away. The API also allows you to attach any backends, services, or applications that you might have and set-up rules that determine how routing is accomplished; it's an API for all your reverse-proxying.

Indeed, for the API: El Guiderino.

The Wave of the Future

Let's Encrypt is due to support wildcard certificates in 2018. At that time Fly will deploy Let's Encrypt within every part of the certificate generation process. At the moment, the * preview hostnames are generated using a wild-card DigiCert. Once wildcard support is released, Fly will make the switch.

We support ACME Version 1; Version 2 is on the horizon. At that time, Fly will be ready to support the new set of standardizations.

As for Fly itself, there are good and nifty things coming. You'll just have to sign-up and see the wonders that await you!

Note: You might be thinking that globe-spanning TLS termination sounds troubling; if you terminate early, what happens to traffic _after termination? It wouldn't be cool to build a service to improve web safety then have traffic bobble around in plaintext after termination within a mystery network. To remedy this, we've got our open source Wormhole. Wormhole creates an encrypted tunnel between two end-points; with Fly and Wormhole, you can have end-to-end encryption without exposing your application to the greater Internet._