Scaling and Fly

Fly Scaling Architecture

Initial Scaling

When you create an application on Fly, it uses our default region plan. This places the application in the best available region for access from the Fly edge network. When an application is created, it is put in the default region, which for most applications will be the closest region to where the application was created.

(The only exception to the "default region" algorithm are Turboku applications which are deployed into the iad region to optimise the Heroku integration).

Scaling Dimensions

There are multiple dimensions of scaling on Fly.

Scaling and configuration

Fly scales within regions by creating more instances of the application as needed. That need is defined by the number of concurrent connections that an application has. The thresholds are defined in the fly.toml file under services.concurrency.

By default, when an application sees 20+ connections, a new instance of the application is started and new connections go to that instance. By adjusting the soft and hard limits of concurrency in the configuration file, you can set how many connections will trigger the creation of a new instance.

Where that instance will appear is down to the regional availability and Fly's auto-balancing mechanisms. These mechanisms decide where a new instance is created.

Regional Scaling

Viewing the Initial Scaling Plan

The command flyctl scale regions will, by default, display the current (or specified) application's scaling plan.

flyctl scale regions
Autoscaling
  Enabled         = true
  Balance Regions = true

This application is using the default scaling plan, so no regions are listed.

Setting the Scaling Plan

Fly gives you direct control of which regions your application instances are in and the mininum number of instances that are kept warm for incoming connections. You can also get a current list of regions with the flyctl platform regions command.

The flyctl scale regions command can also take a list of parameters, made up of a region and a value for a minimum number of instances of the application to be kept running in that region.

flyctl scale regions ams=1 hkg=1 sjc=1
Updating autoscaling config...
Autoscaling
  Enabled         = true
  Balance Regions = true

Regions
  REGION   MIN COUNT   WEIGHT
  ams      1           100
  hkg      1           100
  sjc      1           100

Note that region settings are in addition to existing region settings. To start with a blank slate of region settings, add the flag --reset-all which will clear all existing settings. You can explicitly set a region's count to 0 but this is not the same as returning the region to its default settings. If you want to return a region to its defaults, --reset-region <regionname> will remove the region from the region scale list.

Viewing the Scaling Plan

To view where the instances of a Fly application are currently running, use flyctl status.

flyctl status
App
  Name     = hellofly
  Owner    = dj
  Version  = 299
  Status   = running
  Hostname = hellofly.fly.dev

Deployment Status
  ID          = 59b60abf-ba4f-fb2f-9f78-35a249e2bef5
  Version     = v299
  Status      = successful
  Description = Deployment completed successfully
  Allocations = 3 desired, 3 placed, 3 healthy, 0 unhealthy

Allocations
  ID         VERSION   REGION   DESIRED   STATUS    HEALTH CHECKS   CREATED
  8a9358d1   299       ams      run       running   1 passing       15m36s ago
  7c08ce47   299       nrt      run       running   1 passing       15m36s ago
  1b17a5e6   299       sjc      run       running   1 passing       15m36s ago

These are currently running instances. To view the scaling plan for the application, run flyctl scale regions.

flyctl scale regions
Autoscaling
  Enabled         = true
  Balance Regions = true

Regions
  REGION   MIN COUNT   WEIGHT
  ams      1           100
  hkg      1           100
  sjc      1           100

Notice that the hkg region does not appear in the currently running instances, replaced by nrt (Tokyo). This happens when, for operational reasons, it is temporarily not immediately possible to create a new instance in the desired region. In this case, Fly located a close region and creates the new instance there.

Resetting to Default

The command flyctl scale regions --reset-all will clear all regions back to default, returning to the initial "empty" scale plan. Instances won't be shut down immediately but will be shut down within a few minutes.

Scaling Virtual Machines

Each application instance on Fly runs in its own virtual machine. The number of cores and memory available in the virtual machine can be set for all application instances using the flyctl scale vm command.

Viewing the Current VM Size

Using flyctl scale vm on its own with display the details of the application's current VM sizing.

flyctl scale vm
Size: micro-1x
CPU Cores: 0.125
Memory (GB): 0.125
Memory (MB): 128
Price (Month): 2.67
Price (Second): 1.015e-06

It shows the size name (micro-1x), number of CPU cores, memory (in GB and MB), estimated price per month (if an instance was kept running for a month) and price per second (if an instance was only brought up on demand).

Viewing Available VM Sizes

The flyctl platform vm-sizes command will display the various sizes with cores and memory and current pricing:

flyctl platform vm-sizes
NAME       CPU CORES   MEMORY   PRICE (SECOND)   PRICE (MONTH)
micro-1x   0.12        128 MB   $0.000001        $2.670000
micro-2x   0.25        512 MB   $0.000003        $8.000000
cpu1mem1   1           1 GB     $0.000013        $35.000000
cpu2mem2   2           2 GB     $0.000027        $70.000000
cpu4mem4   4           4 GB     $0.000053        $140.000000
cpu8mem8   8           8 GB     $0.000107        $280.000000

Note: This pricing is correct as of writing (March 2020), run flyctl platform vm-sizes to get the most current pricing.

The CPU Cores column shows how many vCPU cores will be allocated to the virtual machine. Lower than 1, the value reflects the proportion of a shared core that the VM will have available. Greater than 1, it represents the number of cores (from a pool of hyper-threaded cores) that will be available to the VM.

Setting VM Size for an App

Setting the size of the VM is handled by adding the required size name to flyctl scale vm. For example, if we want to double the VM size for our application, from micro-1x to micro-2x, we would run...

flyctl scale vm micro-2x
Scaled VM size to micro-2x
Size: micro-2x
CPU Cores: 0.25
Memory (GB): 0.5
Memory (MB): 512
Price (Month): 8
Price (Second): 3.044e-06

Flyctl responds with the sizes and pricing for a single new instance. All existing instances will be restarted at this new size.