Working with the Machines API
This document covers usage of the Machines REST API to start, stop, update and interact with Fly Machines. For the impatient, flyctl also provides commands for experimenting with the API.
See all possible Machine states in the table below.
Connecting to the API
This guide assumes that you have flyctl
and curl
installed, and have authenticated to Fly.io.
Using the Public api.machines.dev
Endpoint
The easiest (and recommended) way to connect to the Machines API is to use the public api.machines.dev
endpoint, a simpler and more performant alternative to connecting over WireGuard.
Simply skip down to setting up the environment, and make sure to set $FLY_API_HOSTNAME
to https://api.machines.dev
.
Using the Private Machines API Endpoint
You can still access your Machines directly over a Wireguard VPN, and use the private Machines API endpoint: http://_api.internal:4280
. This method requires more setup.
Follow the instructions to set up a permanent WireGuard connection to your Fly.io IPv6 private network. Once you're connected, Fly internal DNS should expose the Machines API endpoint at: http://_api.internal:4280
Connecting via flyctl Proxy
You can also proxy a local port to the internal API endpoint. This section is preserved mainly for the sake of interest, as the public Machines API endpoint is simpler and more performant.
fly machines api-proxy
Pick any organization when asked.
With the above command running, in a separate terminal, try this to confirm you can access the API:
curl http://127.0.0.1:4280
If you successfully reach the API, it should respond with a 404 page not found
error. That's because this was not a defined endpoint.
Setting Up the Environment
Set these environment variables to make the following commands easier to use.
$ export FLY_API_HOSTNAME="https://api.machines.dev" # set to http://_api.internal:4280 when using Wireguard or `http://127.0.0.1:4280` when using 'flyctl proxy'
$ export FLY_API_TOKEN=$(fly auth token)
For local development, you can see the token used by flyctl
with fly auth token
. You can also create a new auth token in the personal access token section of the fly.io dashboard.
In order to access this API on a fly VM, make the token available as a secret:
fly secrets set FLY_API_TOKEN=$(fly auth token)
A convenient way to set the FLY_API_HOSTNAME
is to add it to your Dockerfile
:
ENV FLY_API_HOSTNAME="https://api.machines.dev"
You'll still need to replace the application name and Machine ID for commands to work.
Authentication
All requests must include the Fly API Token in the HTTP Headers as follows:
Authorization: Bearer <fly_api_token>
Create a Fly Application
Machines must be associated with a Fly application. App names must be unique.
curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps" \
-d '{
"app_name": "user-functions",
"org_slug": "personal"
}'
Status: 201
Allocate an IP Address for Global Request Routing
If you intend for Machines to be accessible to the internet, you'll need to allocate an IP address to the application. Currently this is done using flyctl
or the Fly.io GraphQL API. This offers your app automatic, global routing via Anycast. Read more about this in the Networking section.
Example:
fly ips allocate-v4 -a user-functions
TYPE ADDRESS REGION CREATED AT
v4 37.16.9.52 global 7s ago
The app will answer on this IP address, and after a small delay, at user-functions.fly.dev
.
Get Application Details
Get details about an application, like its organization slug and name. Also, to check if the app exists!
curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions"
Status: 200
{
"name": "user-functions",
"organization": {
"name": "My Org",
"slug": "personal"
}
}
Set Application Secrets
For sensitive environment variables, such as credentials, you can set secrets as you would for standard Fly Apps using flyctl
:
flyctl secrets set DATABASE_URL=postgres://example.com/mydb -a user-functions
Machines inherit secrets from the app. Existing Machines must be updated to pick up secrets set after the Machine was created.
For non-sensitive information, you can set different environment variables per Machine at creation time.
Create a Machine
Create a Machine, which starts running immediately. This is where you configure the Machine characteristics, like its CPU and memory. You can also allow connections from the internet through the Fly Proxy. Learn more about this behavior in the networking section.
Following this call, you can make a blocking API request to wait for a Machine to start.
The only required parameter is image
in the config
object. Other parameters:
name
: Unique name for this Machine. If omitted, one is generated for you.
region
: The target region. Omitting this param launches in the same region as your WireGuard peer connection (somewhere near you).
config
: An object defining the Machine configuration. Options:
image
: The Docker image to runguest
: An object with the following options:cpus
: Number of vCPUs (default 1)memory_mb
: Memory in megabytes as multiples of 256 (default 256)kernel_args
: Optional array of strings. Arguments passed to the kernel
auto_destroy
: Optional boolean telling the Machine to destroy itself once it's completesize
: A named size for the VM, e.g.performance-2x
orshared-cpu-2x
. Note:guest
andsize
are mutually exclusive.env
: An object filled with key/value pairs to be set as environment variablesservices
: An array of objects that define a single network service. Check the Machines networking section for more information. Options:protocol
:tcp
orudp
. Learn more about running raw TCP/UDP services.concurrency
: load balancing concurrency settingstype
:connections
(TCP) orrequests
(HTTP). Defaults to connections.soft_limit
: "ideal" service concurrency. We will attempt to spread load to keep services at or below this limithard_limit
: maximum allowed concurrency. We will queue or reject when a service is at this limit
internal_port
: Port the Machines VM listens onports
: An array of objects defining the service's ports and associated handlers. Options:port
: Public-facing port numberhandlers
: Array of connection handlers for TCP-based services.
processes
: An optional array of objects defining multiple processes to run within a VM. The Machine will stop if any process exits without error.name
: Process nameentrypoint
: An array of strings. The process that will runcmd
: An array of strings. The arguments passed to the entrypointenv
: An object filled with key/value pairs to be set as environment variablesuser
: An optional user that the process runs under
schedule
: Optionally one ofhourly
,daily
,weekly
,monthly
. Runs Machine at the given interval. Interval starts at time of Machine creationmounts
: An array of objects that reference previously created persistent volumes. Currently, you may only mount one volume per VM.volume
: The volume ID, visible infly volumes list
, i.e.vol_2n0l3vl60qpv635d
path
: Absolute path on the VM where the volume should be mounted. i.e./data
checks
: An optional object that defines one or more named checks- the key for each check is the check name
- the value for each check supports:
type
:tcp
orhttp
port
: The port to connect to, likely should be the same asinternal_port
interval
: The time between connectivity checkstimeout
: The maximum time a connection can take before being reported as failing its healthcheckmethod
: For http checks, the HTTP method to use to when making the requestpath
: For http checks, the path to send the request toprotocol
: For http checks, whether to usehttp
orhttps
tls_skip_verify
: For http checks with https protocol, whether or not to verify the TLS certificateheaders
: For http checks, an array of objects with string fieldname
and array of strings vieldvalues
An example of two checks:
"checks": {
"tcp-alive": {
"type": "tcp",
"port": 8080,
"interval": "15s",
"timeout": "10s"
},
"http-get": {
"type": "http",
"port": 8080,
"protocol": "http"
"method": "GET",
"path": "/",
"interval": "15s",
"timeout": "10s"
}
}
Example create Machine request and response:
curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines" \
-d '{
"name": "quirky-machine",
"config": {
"image": "flyio/fastify-functions",
"env": {
"APP_ENV": "production"
},
"services": [
{
"ports": [
{
"port": 443,
"handlers": [
"tls",
"http"
]
},
{
"port": 80,
"handlers": [
"http"
]
}
],
"protocol": "tcp",
"internal_port": 8080
}
],
"checks": {
"httpget": {
"type": "http",
"port": 8080,
"method": "GET",
"path": "/",
"interval": "15s",
"timeout": "10s"
}
}
}
}'
Status: 200
{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "starting",
"region": "cdg",
"instance_id": "01G3SHPT434MNW8TS4ENX11RQY",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
"env": {
"APP_ENV": "production"
},
"init": {
"exec": null,
"entrypoint": null,
"cmd": null,
"tty": false
},
"image": "flyio/fastify-functions",
"metadata": null,
"restart": {
"policy": ""
},
"services": [
{
"internal_port": 8080,
"ports": [
{
"handlers": [
"tls",
"http"
],
"port": 443
},
{
"handlers": [
"http"
],
"port": 80
}
],
"protocol": "tcp"
}
],
"guest": {
"cpu_kind": "shared",
"cpus": 1,
"memory_mb": 256
},
"checks": {
"httpget": {
"type": "http",
"port": 8080,
"interval": "15s",
"timeout": "10s",
"method": "GET",
"path": "/"
}
},
"image_ref": {
"registry": "registry-1.docker.io",
"repository": "flyio/fastify-functions",
"tag": "latest",
"digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
"labels": {
}
},
"created_at": "2022-05-23T22:48:21Z"
}
Wait for a Machine to Reach a Specified State
Wait for a Machine to reach a specific state. Specify the desired state with the state
parameter. See the table below for a list of possible states. The default for this parameter is started
.
This request will block for up to 60 seconds. Set a longer timeout with the timeout
parameter.
curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589/wait?instance_id=01GXPEEEOF95AYV5J1HYLGZ8P1&state=stopped"
Status: 200
{
"ok": true
}
Get a Machine
Given a machine ID, fetch details about it.
curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589"
Status: 200
{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "stopped",
"region": "cdg",
"instance_id": "01G3SHPT434MNW8TS4ENX11RQY",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
"env": {
"APP_ENV": "production"
},
"init": {
"exec": null,
"entrypoint": null,
"cmd": null,
"tty": false
},
"image": "flyio/fastify-functions",
"metadata": null,
"restart": {
"policy": ""
},
"services": [
{
"internal_port": 8080,
"ports": [
{
"handlers": [
"tls",
"http"
],
"port": 443
},
{
"handlers": [
"http"
],
"port": 80
}
],
"protocol": "tcp"
}
],
"guest": {
"cpu_kind": "shared",
"cpus": 1,
"memory_mb": 256
}
},
"image_ref": {
"registry": "registry-1.docker.io",
"repository": "flyio/fastify-functions",
"tag": "latest",
"digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
"labels": {
}
},
"created_at": "2022-05-23T22:48:21Z",
"events": [
{
"type": "exit",
"status": "stopped",
"request": {
"exit_event": {
"exit_code": 127,
"exited_at": 1653346105255,
"guest_exit_code": 0,
"guest_signal": -1,
"oom_killed": false,
"requested_stop": false,
"restarting": false,
"signal": -1
},
"restart_count": 0
},
"source": "flyd",
"timestamp": 1653346106636
},
{
"type": "update",
"status": "replacing",
"source": "user",
"timestamp": 1653346105801
},
{
"type": "start",
"status": "started",
"source": "flyd",
"timestamp": 1653346103201
},
{
"type": "launch",
"status": "created",
"source": "user",
"timestamp": 1653346101403
}
]
}
Update a Machine
region
and name
are immutable and cannot be updated. We don't support partial updates, you're required to specify the entire config.
curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589" \
-d '{
"config": {
"image": "flyio/fastify-functions",
"guest": {
"memory_mb": 512,
"cpus": 2,
"cpu_kind": "shared"
},
"env": {
"APP_ENV": "production"
},
"services": [
{
"ports": [
{
"port": 443,
"handlers": [
"tls",
"http"
]
},
{
"port": 80,
"handlers": [
"http"
]
}
],
"protocol": "tcp",
"internal_port": 8080
}
]
}
}'
Status: 200
{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "starting",
"region": "cdg",
"instance_id": "01G3SHPYE8XZ58GD4XRRF9CCKC",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
"env": null,
"init": {
"exec": null,
"entrypoint": null,
"cmd": null,
"tty": false
},
"image": "flyio/fastify-functions",
"metadata": null,
"restart": {
"policy": ""
},
"guest": {
"cpu_kind": "",
"cpus": 2,
"memory_mb": 512
}
},
"image_ref": {
"registry": "registry-1.docker.io",
"repository": "flyio/fastify-functions",
"tag": "latest",
"digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
"labels": {
}
},
"created_at": "2022-05-23T22:48:21Z"
}
Stop a Machine
Stopping a Machine will shut down the VM, but not destroy it. The VM may be started again with machines/<id>/start
.
curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589/stop"
Status: 200
{
"ok": true
}
Start a Machine
Start a previously stopped Machine. Machines that are restarted are completely reset to their original state so that they start clean on the next run.
curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/73d8d46dbee589/start"
Status: 200
{
"previous_state": "stopped"
}
Delete a Machine Permanently
Delete a Machine, never to be seen again. This action cannot be undone!
curl -i -X DELETE \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/24d896dec64879"
Optionally append ?force=true
to the URI to kill a running machine.
Status: 200
{
"ok": true
}
List Machines for an App
curl -i -X GET \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines"
Status: 200
[
{
"id": "73d8d46dbee589",
"name": "quirky-machine",
"state": "started",
"region": "cdg",
"instance_id": "01G3SHPT434MNW8TS4ENX11RQY",
"private_ip": "fdaa:0:3ec2:a7b:5adc:6068:5b85:2",
"config": {
"env": {
"APP_ENV": "production"
},
"init": {
"exec": null,
"entrypoint": null,
"cmd": null,
"tty": false
},
"image": "flyio/fastify-functions:latest",
"metadata": {
},
"restart": {
"policy": "",
"max_retries": 3
},
"size": "shared-cpu-1x"
},
"image_ref": {
"registry": "registry-1.docker.io",
"repository": "flyio/fastify-functions",
"tag": "latest",
"digest": "sha256:e15c11a07e1abbc50e252ac392a908140b199190ab08963b3b5dffc2e813d1e8",
"labels": {
}
},
"created_at": "2022-05-23T22:48:21Z"
}
]
Delete a Fly Application
Machines should be stopped before attempting deletion. Pass force=true
to stop and delete immediately.
curl -i -X DELETE \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions"
Status: 404
{
"error": "Could not find App \"user-functions\""
}
Lease a Machine
Machine leases can be used to obtain an exclusive lock on modifying a Machine.
curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/3d8d413b29d089/lease" \
-d '{
"ttl": 500
}'
ttl
(optional) how long the lease should be held for
Status: 201
{
"status":"success",
"data": {
"nonce":"fed368b018e9",
"expires_at":1666295946,
"owner":"hello@fly.io"
}
}
Status: 409
{
"status":"error",
"message":"machine ID 3d8d413b29d089 lease currently held by hello@fly.io, expires at 2022-10-20 19:59:06 +0000 UTC",
"code":"invalid"
}
How to Use the Provided nonce
Add the fly-machine-lease-nonce
header to all subsequent API calls.
curl -i -X POST \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" -H "fly-machine-lease-nonce: fed368b018e9" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/3d8d413b29d089/stop"
Release the Lease
curl -i -X DELETE \
-H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" -H "fly-machine-lease-nonce: fed368b018e9" \
"${FLY_API_HOSTNAME}/v1/apps/user-functions/machines/3d8d413b29d089/lease"
Notes on Networking
Machines are closed to the public internet by default. To make them accessible via the associated application, you need to:
- Allocate an IP address to the application
- Adding one or more
services
to the Machine config with ports and handlers, as seen in the launch example here
For an application with a single machine, all requests will be routed to that Machine. A Machine in the stopped
state will be started up automatically when a request arrives.
For an application with multiple Machines with the same configuration, requests will be distributed across them. Warm-start behavior in this situation is not well-defined now, so should not be relied upon for apps with multiple Machines.
Requests to Machines with mixed configurations will be distributed across Machines whos configurations match the request. For example, if 3 out of 6 Machines have service configurations set to listen on port 80, requests to port 80 will be distributed amongst those 3.
Reaching Machines
Machines can be reached within the private network by hostname in the format <id>.vm.<app-name>.internal
. For example: 3d8d413b29d089.vm.user-functions.internal
.
Machine States
This table explains the possible Machine states. A Machine may only be in one state at a time.
created | Initial status |
starting | Transitioning from stopped to started |
started | Running and network-accessible |
stopping | Transitioning from started to stopped |
stopped | Exited, either on its own or explicitly stopped |
replacing | User-initiated configuration change (image, VM size, etc.) in progress |
destroying | User asked for the machine to be completely removed |
destroyed | No longer exists |
Internal note: the replaced state is only possible when requesting a specific instance_id
.