Fly GPUs

Fly.io has GPUs! If you have workloads that would benefit from GPU acceleration, Fly GPU Machines may be for you.

What can I use Fly GPUs for?

Three models of GPU are available: NVIDIA A100 40G PCIe, A100 80G SXM, and L40S.

A100 units are all about the tensor cores, and are positioned for inference, model training, and intensive high-precision computation tasks like scientific simulations. As their names suggest, they have 40GB and 80GB of GPU memory. (A100 datasheet)

The L40S cards are all-rounders; they’ve got tensor cores, RT cores, and NVENC/NVDEC, and have 48GB of GPU RAM. Choose the L40S to accelerate graphics or video workloads, as well as for inference. (L40S datasheet)

Right now each Fly GPU Machine uses a single full GPU. A single GPU is well suited to rendering, encoding/decoding, inference, and a smidgen of fine tuning. Training large models from scratch requires much, much beefier resources.

Go to the GPU Quickstart to get off the ground fast, or read more practicalities in Getting started with Fly GPUs.

Regions with GPUs

Currently GPUs are available in the following regions:

• l40s: ord
• a100-40gb: ord
• a100-80gb: ams, iad, mia, sjc, syd

Examples

Here’s some more inspiration for your GPU Machines project:

• Python GPU Dev Machine
• Elixir Llama2-13b on Fly.io GPUs
• Fly.io CUDA example
• Deploying CLIP on Fly.io
• GitHub fly-apps repos with the gpu topic