Machines

Fly Machines are Firecracker VMs with a fast REST API that can boot instances in about 300ms, in any region supported by Fly.io.

The Machines API gives you efficient, low-level control over VM provisioning, supported by Fly.io infrastructure and networking features.

Machines are also the spawning ground for new platform features like wake-on-request (also known as scale-to-zero). You can stop a running machine to save on compute costs. It then may be started automatically when a request arrives at the Fly proxy.

This document covers usage of the Machines REST API to start, stop, update and interact with machines. For the impatient, flyctl also provides commands for experimenting with the API.

See all possible machine states in the table below.

Connecting to the API

The Machines API endpoint requires a connection to your Fly private network - either by running on a VM inside the network, or via a WireGuard VPN, or using flyctl proxy.

This guide assumes that you have flyctl and curl installed, and have authenticated to the Fly.io platform.

Connecting via WireGuard

This method requires more setup but gives you direct access to machines over the Wireguard VPN.

First, follow the instructions to set up a permanent WireGuard connection to your Fly network. We recommend WireGuard because you can directly test your machines from your local machine.

Once you're connected, Fly internal DNS should expose the Machines API endpoint at: http://_api.internal:4280

Connecting via flyctl Proxy

For quick testing of this API, you can proxy a local port to the internal API endpoint. Pick any organization when asked.

flyctl machines api-proxy

With the above command running, in a separate terminal, try this to confirm you can access the API:

curl http://127.0.0.1:4280

If you successfully reach the API, it should respond with a 404 page not found error. That's because this was not a defined endpoint.

Setting Up the Environment

If you are using a Wireguard VPN or a flyctl proxy, set these environment variables to make the following commands easier to use.

$ export FLY_API_HOSTNAME="_api.internal:4280" # or set to `127.0.0.1:4280` when using 'flyctl proxy'
$ export FLY_API_TOKEN=$(fly auth token)

In order to access this API on a fly VM, make the token available as a secret:

flyctl secrets set FLY_API_TOKEN=$(fly auth token)

A convenient way to set the FLYAPIHOSTNAME is to add it to your Dockerfile:

ENV FLY_API_HOSTNAME="_api.internal:4280

You'll still need to replace the application name and machine ID for commands to work.

Authentication

All requests must include the the fly API Token in the HTTP Headers as follows:

Authorization: Bearer <fly_api_token>

Create a Fly Application

Machines must be associated with a Fly application. App names must be unique.

curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"http://${FLY_API_HOSTNAME}/v1/apps" \
-d '{
  "app_name": "user-functions",
  "org_slug": "personal"
}'

Status: 201

Allocate an IP Address for Global Request Routing

If you intend for machines to be accessible to the internet, you'll need to allocate an IP address to the application. Currently this is done using flyctl or the Fly GraphQL API. This offers your app automatic, global routing via Anycast. Read more about this in the Networking section.

Example:

flyctl ips allocate-v4 -a user-functions
TYPE ADDRESS    REGION CREATED AT
v4   37.16.9.52 global 7s ago

The app will answer on this IP address, and after a small delay, at user-functions.fly.dev.

Get Application Details

Get details about an application, like its organization slug and name. Also, to check if the app exists!

curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"http://${FLY_API_HOSTNAME}/v1/apps/user-functions"

Status: 200

{
"name": "user-functions",
"organization": {
"name": "My Org",
"slug": "personal"
}
}

Set Application Secrets

For sensitive environment variables, such as credentials, you can set secrets as you would for standard Fly apps using flyctl:

flyctl secrets set DATABASE_URL=postgres://example.com/mydb -a user-functions

Machines inherit secrets from the app. Existing machines must be updated to pick up secrets set after the machine was created.

For non-sensitive information, you can set different environment variables per machine at creation time.

Create and Start a Machine

Create and start a machine immediately. This is where you configure the machine characteristics, like its CPU and memory. You can also allow connections from the internet through the Fly proxy. Learn more about this behavior in the networking section.

Following this call, you can make a blocking API request to wait for a machine to start.

The only required parameter is image in the config object. Other parameters:

name: Unique name for this machine. If omitted, one is generated for you.

region: The target region. Omitting this param launches in the same region as your WireGuard peer connection (somewhere near you).

config: An object defining the machine configuration. Options

  • image: The Docker image to run
  • guest: An object with the following options:

    • cpus: Number of vCPUs (default 1)
    • memory_mb: Memory in megabytes as multiples of 256 (default 256)
  • env: An object filled with key/value pairs to be set as environment variables

  • services: An array of objects that define a single network service. Check the machines networking section for more information. Options:

    • protocol: tcp or udp. Learn more about running raw TCP/UDP services.
    • concurrency: load balancing concurrency settings
      • type: connections (TCP) or requests (HTTP). Defaults to connections.
      • soft_limit: "ideal" service concurrency. We will attempt to spread load to keep services at or below this limit
      • hard_limit: maximum allowed concurrency. We will queue or reject when a service is at this limit
    • internal_port: Port the machine VM listens on
    • ports: An array of objects defining the service's ports and associated handlers. Options:
      • port: Public-facing port number
      • handlers: Array of connection handlers for TCP-based services.
  • mounts: An array of objects that reference previously created persistent volumes. Currently, you may only mount one volume per VM.

    • volume: The volume ID, visible in fly volumes list, i.e. vol_2n0l3vl60qpv635d
    • path: Absolute path on the VM where the volume should be mounted. i.e. /data
curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"http://${FLY_API_HOSTNAME}/v1/apps/user-functions/machines" \
-d '{
  "name": "quirky-machine",
  "config": {
    "image": "flyio/fastify-functions",
    "env": {
      "APP_ENV": "production"
    },
    "services": [
      {
        "ports": [
          {
            "port": 443,
            "handlers": [
              "tls",
              "http"
            ]
          },
          {
            "port": 80,
            "handlers": [
              "http"
            ]
          }
        ],
        "protocol": "tcp",
        "internal_port": 8080
      }
    ]
  }
}'

Status: 200

{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "starting",
"region": "cdg",
"instance_id": "01G3SHPT434MNW8TS4ENX11RQY",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
"env": {
  "APP_ENV": "production"
},
"init": {
  "exec": null,
  "entrypoint": null,
  "cmd": null,
  "tty": false
},
"image": "flyio/fastify-functions",
"metadata": null,
"restart": {
  "policy": ""
},
"services": [
  {
    "internal_port": 8080,
    "ports": [
      {
        "handlers": [
          "tls",
          "http"
        ],
        "port": 443
      },
      {
        "handlers": [
          "http"
        ],
        "port": 80
      }
    ],
    "protocol": "tcp"
  }
],
"guest": {
  "cpu_kind": "shared",
  "cpus": 1,
  "memory_mb": 256
}
},
"image_ref": {
"registry": "registry-1.docker.io",
"repository": "flyio/fastify-functions",
"tag": "latest",
"digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
"labels": {
}
},
"created_at": "2022-05-23T22:48:21Z"
}

Wait for a Machine to Reach a Specified State

Wait for a machine to reach a specific state. Specify the desired state with the state parameter. See the table below for a list of possible states. The default for this parameter is started.

This request will block for up to 60 seconds. Set a longer timeout with the timeout parameter.

curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"http://${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589/wait"

Status: 200

{
"ok": true
}

Get a Machine

Given a machine ID, fetch details about it.

curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"http://${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589"

Status: 200

{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "stopped",
"region": "cdg",
"instance_id": "01G3SHPT434MNW8TS4ENX11RQY",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
"env": {
  "APP_ENV": "production"
},
"init": {
  "exec": null,
  "entrypoint": null,
  "cmd": null,
  "tty": false
},
"image": "flyio/fastify-functions",
"metadata": null,
"restart": {
  "policy": ""
},
"services": [
  {
    "internal_port": 8080,
    "ports": [
      {
        "handlers": [
          "tls",
          "http"
        ],
        "port": 443
      },
      {
        "handlers": [
          "http"
        ],
        "port": 80
      }
    ],
    "protocol": "tcp"
  }
],
"guest": {
  "cpu_kind": "shared",
  "cpus": 1,
  "memory_mb": 256
}
},
"image_ref": {
"registry": "registry-1.docker.io",
"repository": "flyio/fastify-functions",
"tag": "latest",
"digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
"labels": {
}
},
"created_at": "2022-05-23T22:48:21Z",
"events": [
{
  "type": "exit",
  "status": "stopped",
  "request": {
    "exit_event": {
      "exit_code": 127,
      "exited_at": 1653346105255,
      "guest_exit_code": 0,
      "guest_signal": -1,
      "oom_killed": false,
      "requested_stop": false,
      "restarting": false,
      "signal": -1
    },
    "restart_count": 0
  },
  "source": "flyd",
  "timestamp": 1653346106636
},
{
  "type": "update",
  "status": "replacing",
  "source": "user",
  "timestamp": 1653346105801
},
{
  "type": "start",
  "status": "started",
  "source": "flyd",
  "timestamp": 1653346103201
},
{
  "type": "launch",
  "status": "created",
  "source": "user",
  "timestamp": 1653346101403
}
]
}

Update a Machine

region and name are immutable and cannot be updated. Updating the config requires specifying the complete config.

curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"http://${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589" \
-d '{
  "config": {
    "image": "flyio/fastify-functions",
    "guest": {
      "memory_mb": 512,
      "cpus": 2,
      "cpu_kind": "shared"
    },
    "env": {
      "APP_ENV": "production"
    },
    "services": [
      {
        "ports": [
          {
            "port": 443,
            "handlers": [
              "tls",
              "http"
            ]
          },
          {
            "port": 80,
            "handlers": [
              "http"
            ]
          }
        ],
        "protocol": "tcp",
        "internal_port": 8080
      }
    ]
  }
}'

Status: 200

{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "starting",
"region": "cdg",
"instance_id": "01G3SHPYE8XZ58GD4XRRF9CCKC",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
"env": null,
"init": {
  "exec": null,
  "entrypoint": null,
  "cmd": null,
  "tty": false
},
"image": "flyio/fastify-functions",
"metadata": null,
"restart": {
  "policy": ""
},
"guest": {
  "cpu_kind": "",
  "cpus": 2,
  "memory_mb": 512
}
},
"image_ref": {
"registry": "registry-1.docker.io",
"repository": "flyio/fastify-functions",
"tag": "latest",
"digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
"labels": {
}
},
"created_at": "2022-05-23T22:48:21Z"
}

Stop a Machine

Stopping a machine will shut down the VM, but not destroy it. The VM may be started again with machines/start.

curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"http://${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589/stop"

Status: 200

{
"ok": true
}

Start a Machine

Start a previously stopped machine. Machines that are restarted are completely reset to their original state so that they start clean on the next run.

curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"http://${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589/start"

Status: 200

{
"previous_state": "stopped"
}

Delete a Machine Permanently

Delete a machine, never to be seen again. This action cannot be undone!

curl -i -X DELETE \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"http://${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/24d896dec64879"

Status: 200

{
"ok": true
}

List Machines for an App

curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"http://${FLY_API_HOSTNAME}/v1/apps/user-functions/machines"

Status: 200

[
{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "started",
"region": "cdg",
"instance_id": "01G3SHPT434MNW8TS4ENX11RQY",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
  "env": {
    "APP_ENV": "production"
  },
  "init": {
    "exec": null,
    "entrypoint": null,
    "cmd": null,
    "tty": false
  },
  "image": "flyio/fastify-functions:latest",
  "metadata": {
  },
  "restart": {
    "policy": "",
    "max_retries": 3
  },
  "size": "shared-cpu-1x"
},
"image_ref": {
  "registry": "registry-1.docker.io",
  "repository": "flyio/fastify-functions",
  "tag": "latest",
  "digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
  "labels": {
  }
},
"created_at": "2022-05-23T22:48:21Z"
}
]

Delete a Fly Application

Machines should be stopped before attempting deletion. Pass force=true to stop and delete immediately.

curl -i -X DELETE \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"http://${FLY_API_HOSTNAME}/v1/apps/user-functions"

Status: 404

{
"error": "Could not find App \"user-functions\""
}

Notes on Networking

Machines are closed to the public internet by default. To make them accessible via the associated application, you need to:

  • Allocate an IP address to the application
  • Adding one or more services to the machine config with ports and handlers, as seen in the launch example here

For an application with a single machine, all requests will be routed to that machine. A machine in the stopped state will be started up automatically when a request arrives.

For an application with multiple machines with the same configuration, requests will be distributed across them. Warm-start behavior in this situation is not well-defined now, so should not be relied upon for apps with multiple machines.

Machine States

This table explains the possible machine states. A machine may only be in one state at a time.

created Initial status
started Running and network-accessible
stopping Transitioning from started to stopped
stopped Exited, either on its own or explicitly stopped
replacing User-initiated configuration change (image, VM size, etc.) in progress
destroying User asked for the machine to be completely removed
destroyed No longer exists

Internal note: the replaced state is only possible when requesting a specific instance_id.