Working with the Machines API

This document covers usage of the Machines REST API to start, stop, update and interact with Fly Machines. For the impatient, flyctl also provides commands for experimenting with the API.

See all possible Machine states in the table below.

Connecting to the API

This guide assumes that you have flyctl and curl installed, and have authenticated to Fly.io.

Using the Public api.machines.dev Endpoint

The easiest (and recommended) way to connect to the Machines API is to use the public api.machines.dev endpoint, a simpler and more performant alternative to connecting over WireGuard.

Simply skip down to setting up the environment, and make sure to set $FLY_API_HOSTNAME to https://api.machines.dev.

Using the Private Machines API Endpoint

You can still access your Machines directly over a Wireguard VPN, and use the private Machines API endpoint: http://_api.internal:4280. This method requires more setup.

Follow the instructions to set up a permanent WireGuard connection to your Fly.io IPv6 private network. Once you're connected, Fly internal DNS should expose the Machines API endpoint at: http://_api.internal:4280

Connecting via flyctl Proxy

You can also proxy a local port to the internal API endpoint. This section is preserved mainly for the sake of interest, as the public Machines API endpoint is simpler and more performant.

fly machines api-proxy

Pick any organization when asked.

With the above command running, in a separate terminal, try this to confirm you can access the API:

curl http://127.0.0.1:4280

If you successfully reach the API, it should respond with a 404 page not found error. That's because this was not a defined endpoint.

Setting Up the Environment

Set these environment variables to make the following commands easier to use.

$ export FLY_API_HOSTNAME="https://api.machines.dev" # set to http://_api.internal:4280 when using Wireguard or `http://127.0.0.1:4280` when using 'flyctl proxy'
$ export FLY_API_TOKEN=$(fly auth token)

For local development, you can see the token used by flyctl with fly auth token. You can also create a new auth token in the personal access token section of the fly.io dashboard.

In order to access this API on a fly VM, make the token available as a secret:

fly secrets set FLY_API_TOKEN=$(fly auth token)

A convenient way to set the FLY_API_HOSTNAME is to add it to your Dockerfile:

ENV FLY_API_HOSTNAME="https://api.machines.dev"

You'll still need to replace the application name and Machine ID for commands to work.

Authentication

All requests must include the Fly API Token in the HTTP Headers as follows:

Authorization: Bearer <fly_api_token>

Create a Fly Application

Machines must be associated with a Fly application. App names must be unique.

curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps" \
-d '{
  "app_name": "user-functions",
  "org_slug": "personal"
}'

Status: 201

Allocate an IP Address for Global Request Routing

If you intend for Machines to be accessible to the internet, you'll need to allocate an IP address to the application. Currently this is done using flyctl or the Fly.io GraphQL API. This offers your app automatic, global routing via Anycast. Read more about this in the Networking section.

Example:

fly ips allocate-v4 -a user-functions

TYPE ADDRESS    REGION CREATED AT
v4   37.16.9.52 global 7s ago

The app will answer on this IP address, and after a small delay, at user-functions.fly.dev.

Get Application Details

Get details about an application, like its organization slug and name. Also, to check if the app exists!

curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions"

Status: 200

{
"name": "user-functions",
"organization": {
"name": "My Org",
"slug": "personal"
}
}

Set Application Secrets

For sensitive environment variables, such as credentials, you can set secrets as you would for standard Fly Apps using flyctl:

flyctl secrets set DATABASE_URL=postgres://example.com/mydb -a user-functions

Machines inherit secrets from the app. Existing Machines must be updated to pick up secrets set after the Machine was created.

For non-sensitive information, you can set different environment variables per Machine at creation time.

Create a Machine

Create a Machine, which starts running immediately. This is where you configure the Machine characteristics, like its CPU and memory. You can also allow connections from the internet through the Fly Proxy. Learn more about this behavior in the networking section.

Following this call, you can make a blocking API request to wait for a Machine to start.

The only required parameter is image in the config object. Other parameters:

name: Unique name for this Machine. If omitted, one is generated for you.

region: The target region. Omitting this param launches in the same region as your WireGuard peer connection (somewhere near you).

config: An object defining the Machine configuration. Options:

• image: The Docker image to run
• guest: An object with the following options:
  • cpus: Number of vCPUs (default 1)
  • memory_mb: Memory in megabytes as multiples of 256 (default 256)
  • kernel_args: Optional array of strings. Arguments passed to the kernel

• auto_destroy: Optional boolean telling the Machine to destroy itself once it's complete

• size: A named size for the VM, e.g. performance-2x or shared-cpu-2x. Note: guest and size are mutually exclusive.

• env: An object filled with key/value pairs to be set as environment variables

• services: An array of objects that define a single network service. Check the Machines networking section for more information. Options:

  • protocol: tcp or udp. Learn more about running raw TCP/UDP services.
  • concurrency: load balancing concurrency settings
    • type: connections (TCP) or requests (HTTP). Defaults to connections.
    • soft_limit: "ideal" service concurrency. We will attempt to spread load to keep services at or below this limit
    • hard_limit: maximum allowed concurrency. We will queue or reject when a service is at this limit
  • internal_port: Port the Machines VM listens on
  • ports: An array of objects defining the service's ports and associated handlers. Options:
    • port: Public-facing port number
    • handlers: Array of connection handlers for TCP-based services.

• processes: An optional array of objects defining multiple processes to run within a VM. The Machine will stop if any process exits without error.

  • name: Process name
  • entrypoint: An array of strings. The process that will run
  • cmd: An array of strings. The arguments passed to the entrypoint
  • env: An object filled with key/value pairs to be set as environment variables
  • user: An optional user that the process runs under

• schedule: Optionally one of hourly, daily, weekly, monthly. Runs Machine at the given interval. Interval starts at time of Machine creation

• mounts: An array of objects that reference previously created persistent volumes. Currently, you may only mount one volume per VM.

  • volume: The volume ID, visible in fly volumes list, i.e. vol_2n0l3vl60qpv635d
  • path: Absolute path on the VM where the volume should be mounted. i.e. /data

• checks: An optional object that defines one or more named checks

  • the key for each check is the check name
  • the value for each check supports:
    • type: tcp or http
    • port: The port to connect to, likely should be the same as internal_port
    • interval: The time between connectivity checks
    • timeout: The maximum time a connection can take before being reported as failing its healthcheck
    • method: For http checks, the HTTP method to use to when making the request
    • path: For http checks, the path to send the request to
    • protocol: For http checks, whether to use http or https
    • tls_skip_verify: For http checks with https protocol, whether or not to verify the TLS certificate
    • headers: For http checks, an array of objects with string field name and array of strings vield values

An example of two checks:

        "checks": {
            "tcp-alive": {
                "type": "tcp",
                "port": 8080,
                "interval": "15s",
                "timeout": "10s"
            },
            "http-get": {
                "type": "http",
                "port": 8080,
                "protocol": "http"
                "method": "GET",
                "path": "/",
                "interval": "15s",
                "timeout": "10s"
            }
        }

Example create Machine request and response:

curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines" \
-d '{
  "name": "quirky-machine",
  "config": {
    "image": "flyio/fastify-functions",
    "env": {
      "APP_ENV": "production"
    },
    "services": [
      {
        "ports": [
          {
            "port": 443,
            "handlers": [
              "tls",
              "http"
            ]
          },
          {
            "port": 80,
            "handlers": [
              "http"
            ]
          }
        ],
        "protocol": "tcp",
        "internal_port": 8080
      }
    ],
    "checks": {
        "httpget": {
            "type": "http",
            "port": 8080,
            "method": "GET",
            "path": "/",
            "interval": "15s",
            "timeout": "10s"
        }
    }
  }
}'

Status: 200

{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "starting",
"region": "cdg",
"instance_id": "01G3SHPT434MNW8TS4ENX11RQY",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
"env": {
  "APP_ENV": "production"
},
"init": {
  "exec": null,
  "entrypoint": null,
  "cmd": null,
  "tty": false
},
"image": "flyio/fastify-functions",
"metadata": null,
"restart": {
  "policy": ""
},
"services": [
  {
    "internal_port": 8080,
    "ports": [
      {
        "handlers": [
          "tls",
          "http"
        ],
        "port": 443
      },
      {
        "handlers": [
          "http"
        ],
        "port": 80
      }
    ],
    "protocol": "tcp"
  }
],
"guest": {
  "cpu_kind": "shared",
  "cpus": 1,
  "memory_mb": 256
},
"checks": {
  "httpget": {
    "type": "http",
    "port": 8080,
    "interval": "15s",
    "timeout": "10s",
    "method": "GET",
    "path": "/"
  }
},
"image_ref": {
"registry": "registry-1.docker.io",
"repository": "flyio/fastify-functions",
"tag": "latest",
"digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
"labels": {
}
},
"created_at": "2022-05-23T22:48:21Z"
}

Wait for a Machine to Reach a Specified State

Wait for a Machine to reach a specific state. Specify the desired state with the state parameter. See the table below for a list of possible states. The default for this parameter is started.

This request will block for up to 60 seconds. Set a longer timeout with the timeout parameter.

curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589/wait?instance_id=01GXPEEEOF95AYV5J1HYLGZ8P1&state=stopped"

Status: 200

{
"ok": true
}

Get a Machine

Given a machine ID, fetch details about it.

curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589"

Status: 200

{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "stopped",
"region": "cdg",
"instance_id": "01G3SHPT434MNW8TS4ENX11RQY",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
"env": {
  "APP_ENV": "production"
},
"init": {
  "exec": null,
  "entrypoint": null,
  "cmd": null,
  "tty": false
},
"image": "flyio/fastify-functions",
"metadata": null,
"restart": {
  "policy": ""
},
"services": [
  {
    "internal_port": 8080,
    "ports": [
      {
        "handlers": [
          "tls",
          "http"
        ],
        "port": 443
      },
      {
        "handlers": [
          "http"
        ],
        "port": 80
      }
    ],
    "protocol": "tcp"
  }
],
"guest": {
  "cpu_kind": "shared",
  "cpus": 1,
  "memory_mb": 256
}
},
"image_ref": {
"registry": "registry-1.docker.io",
"repository": "flyio/fastify-functions",
"tag": "latest",
"digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
"labels": {
}
},
"created_at": "2022-05-23T22:48:21Z",
"events": [
{
  "type": "exit",
  "status": "stopped",
  "request": {
    "exit_event": {
      "exit_code": 127,
      "exited_at": 1653346105255,
      "guest_exit_code": 0,
      "guest_signal": -1,
      "oom_killed": false,
      "requested_stop": false,
      "restarting": false,
      "signal": -1
    },
    "restart_count": 0
  },
  "source": "flyd",
  "timestamp": 1653346106636
},
{
  "type": "update",
  "status": "replacing",
  "source": "user",
  "timestamp": 1653346105801
},
{
  "type": "start",
  "status": "started",
  "source": "flyd",
  "timestamp": 1653346103201
},
{
  "type": "launch",
  "status": "created",
  "source": "user",
  "timestamp": 1653346101403
}
]
}

Update a Machine

region and name are immutable and cannot be updated. We don't support partial updates, you're required to specify the entire config.

curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589" \
-d '{
  "config": {
    "image": "flyio/fastify-functions",
    "guest": {
      "memory_mb": 512,
      "cpus": 2,
      "cpu_kind": "shared"
    },
    "env": {
      "APP_ENV": "production"
    },
    "services": [
      {
        "ports": [
          {
            "port": 443,
            "handlers": [
              "tls",
              "http"
            ]
          },
          {
            "port": 80,
            "handlers": [
              "http"
            ]
          }
        ],
        "protocol": "tcp",
        "internal_port": 8080
      }
    ]
  }
}'

Status: 200

{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "starting",
"region": "cdg",
"instance_id": "01G3SHPYE8XZ58GD4XRRF9CCKC",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
"env": null,
"init": {
  "exec": null,
  "entrypoint": null,
  "cmd": null,
  "tty": false
},
"image": "flyio/fastify-functions",
"metadata": null,
"restart": {
  "policy": ""
},
"guest": {
  "cpu_kind": "",
  "cpus": 2,
  "memory_mb": 512
}
},
"image_ref": {
"registry": "registry-1.docker.io",
"repository": "flyio/fastify-functions",
"tag": "latest",
"digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
"labels": {
}
},
"created_at": "2022-05-23T22:48:21Z"
}

Stop a Machine

Stopping a Machine will shut down the VM, but not destroy it. The VM may be started again with machines/<id>/start.

curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589/stop"

Status: 200

{
"ok": true
}

Start a Machine

Start a previously stopped Machine. Machines that are restarted are completely reset to their original state so that they start clean on the next run.

curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589/start"

Status: 200

{
"previous_state": "stopped"
}

Delete a Machine Permanently

Delete a Machine, never to be seen again. This action cannot be undone!

curl -i -X DELETE \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/24d896dec64879" 

Optionally append ?force=true to the URI to kill a running machine.

Status: 200

{
"ok": true
}

List Machines for an App

curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines"

Status: 200

[
{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "started",
"region": "cdg",
"instance_id": "01G3SHPT434MNW8TS4ENX11RQY",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
  "env": {
    "APP_ENV": "production"
  },
  "init": {
    "exec": null,
    "entrypoint": null,
    "cmd": null,
    "tty": false
  },
  "image": "flyio/fastify-functions:latest",
  "metadata": {
  },
  "restart": {
    "policy": "",
    "max_retries": 3
  },
  "size": "shared-cpu-1x"
},
"image_ref": {
  "registry": "registry-1.docker.io",
  "repository": "flyio/fastify-functions",
  "tag": "latest",
  "digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
  "labels": {
  }
},
"created_at": "2022-05-23T22:48:21Z"
}
]

Delete a Fly Application

Machines should be stopped before attempting deletion. Pass force=true to stop and delete immediately.

curl -i -X DELETE \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions"

Status: 404

{
"error": "Could not find App \"user-functions\""
}

Lease a Machine

Machine leases can be used to obtain an exclusive lock on modifying a Machine.

curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/3d8d413b29d089/lease" \
-d '{
"ttl": 500
}'

• ttl (optional) how long the lease should be held for

Status: 201

{
"status":"success",
"data": {
"nonce":"fed368b018e9",
"expires_at":1666295946,
"owner":"hello@fly.io"
}
}

Status: 409

{
"status":"error",
"message":"machine ID 3d8d413b29d089 lease currently held by hello@fly.io, expires at 2022-10-20 19:59:06 +0000 UTC",
"code":"invalid"
}

How to Use the Provided nonce

Add the fly-machine-lease-nonce header to all subsequent API calls.

curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" -H "fly-machine-lease-nonce: fed368b018e9" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/3d8d413b29d089/stop"

Release the Lease

curl -i -X DELETE \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" -H "fly-machine-lease-nonce: fed368b018e9" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/3d8d413b29d089/lease" 

Notes on Networking

Machines are closed to the public internet by default. To make them accessible via the associated application, you need to:

• Allocate an IP address to the application
• Adding one or more services to the Machine config with ports and handlers, as seen in the launch example here

For an application with a single machine, all requests will be routed to that Machine. A Machine in the stopped state will be started up automatically when a request arrives.

For an application with multiple Machines with the same configuration, requests will be distributed across them. Warm-start behavior in this situation is not well-defined now, so should not be relied upon for apps with multiple Machines.

Requests to Machines with mixed configurations will be distributed across Machines whos configurations match the request. For example, if 3 out of 6 Machines have service configurations set to listen on port 80, requests to port 80 will be distributed amongst those 3.

Reaching Machines

Machines can be reached within the private network by hostname in the format <id>.vm.<app-name>.internal. For example: 3d8d413b29d089.vm.user-functions.internal.

Machine States

This table explains the possible Machine states. A Machine may only be in one state at a time.

Internal note: the replaced state is only possible when requesting a specific instance_id.