<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/">
  <title>Infra Log</title>
  <subtitle>The home for Infra Engineering content on Fly.io.</subtitle>
  <id>https://fly.io/infra-log/</id>
  <link href="https://fly.io/infra-log/"/>
  <link href="https://fly.io/infra-log/" rel="self"/>
  <updated>2026-11-04T00:00:00+00:00</updated>
  <author>
    <name>Fly</name>
  </author>
  <entry>
    <title>Managed Postgres control plane in IAD went read-only after etcd hit its space quota</title>
    <link rel="alternate" href="https://fly.io/infra-log/mpg-control-plane-read-only/"/>
    <id>https://fly.io/infra-log/mpg-control-plane-read-only/</id>
    <published>2026-11-04T00:00:00+00:00</published>
    <updated>2026-11-06T04:10:35+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;For a couple hours, the Managed Postgres (MPG) control plane in IAD couldn&amp;rsquo;t accept writes. Customer databases kept running and serving traffic the whole time, but control plane operations like resizing a cluster or changing its plan failed. A handful of clusters that were mid-maintenance sat stuck in a &lt;code&gt;reapplying&lt;/code&gt; state until we cleared the problem.&lt;/p&gt;

&lt;p&gt;The MPG control plane keeps its orchestration state in etcd. A scheduled maintenance run reapplied configuration to 170 IAD clusters over about 40 minutes. Every reconcile rewrites the cluster&amp;rsquo;s full snapshot and activity payloads, roughly 130 KB each time. etcd is a multi-version store, and our periodic compaction keeps about an hour of old revisions around, so all of those rewrites piled up instead of replacing each other. The maintenance wave added about 0.7 GB on top of 1.33 GB of live data, which pushed the database past its 2 GiB &lt;code&gt;quota-backend-bytes&lt;/code&gt; limit. When that happens, etcd raises a &lt;code&gt;NOSPACE&lt;/code&gt; alarm and refuses further writes, so everything after that returned &lt;code&gt;etcdserver: mvcc: database space exceeded&lt;/code&gt;. The rest of the maintenance run failed, workflow locks and worker claims failed, and four clusters were left mid-update.&lt;/p&gt;

&lt;p&gt;Our existing etcd alerts weren&amp;rsquo;t enough. We only noticed after the maintenance run&amp;rsquo;s circuit breaker had tripped.&lt;/p&gt;

&lt;p&gt;To recover, we compacted etcd, which brought in-use data down to about 1.26 GB. Compaction alone doesn&amp;rsquo;t shrink the file on disk, so the next step was a defrag. Our first defrag attempt failed with &lt;code&gt;ENOSPC&lt;/code&gt; because defrag builds a temporary copy of the database and the 3 GB volumes didn&amp;rsquo;t have room for it. We extended the volumes online to 10 GB, defragged each member (followers first, moving leadership off the leader before doing it last), and the IAD database dropped from 2.1 GB to 1.1 GB. Once we disarmed the alarm, the control plane started accepting writes again and the stuck clusters finished their updates on their own. FRA was sitting at about 90% of its quota, so we did the same there (1.9 GB down to 315 MB) and raised the quota in both IAD and FRA to 8 GiB.&lt;/p&gt;

&lt;p&gt;One detail made this worse than it needed to be. We offload cluster history from etcd to object storage, but that offload only runs when new history is written. Most clusters hadn&amp;rsquo;t been offloaded, and once etcd went read-only, no new history could be written, so nothing could be offloaded either.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Petsem was flaky for a bit</title>
    <link rel="alternate" href="https://fly.io/infra-log/petsem-was-flaky/"/>
    <id>https://fly.io/infra-log/petsem-was-flaky/</id>
    <published>2026-11-04T00:00:00+00:00</published>
    <updated>2026-11-06T04:10:35+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;Our secrets store, Petsem, runs on a SQLite primary in IAD. We’re exploring a change away from SQLite, which may or may not happen, but which requires more record tracking in the database. Some time after being deployed, the extra write volume from this change tipped the database into sustained lock contention, and a little over half of write requests started failing. Replicas, and reads from the primary, were not affected. We built and released a new version of Petsem without the tracking code, which stopped the intermittent errors, and we’re gonna stare really hard at whatever our next approach is here.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Expired TLS certificate on sprites.app</title>
    <link rel="alternate" href="https://fly.io/infra-log/expired-tls-certificate/"/>
    <id>https://fly.io/infra-log/expired-tls-certificate/</id>
    <published>2026-10-23T00:00:00+00:00</published>
    <updated>2026-10-28T18:12:58+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;Hosted Sprites on &lt;code&gt;*.sprites.app&lt;/code&gt; were unavailable for roughly 75 minutes, due to a bug in our certificate provisioning meeting a bug in the Fly Proxy.&lt;/p&gt;

&lt;p&gt;We issue both RSA and ECDSA certs for domains via Let’s Encrypt, and renew both of these in lockstep. When the certs came up for renewal about a month before the outage, the RSA cert renewed successfully but the ECDSA renewal either silently failed or never ran. We’d since found and fixed this bug, but the Sprites cert was renewed before the fix shipped, leaving it with a fresh RSA cert and a soon-to-expire ECDSA cert.&lt;/p&gt;

&lt;p&gt;When the ECDSA cert expired, our proxy &lt;em&gt;should&lt;/em&gt; have fallen back to the still-valid RSA cert. That fallback worked well for errors and absent certs, but when it “successfully” found an expired cert it bailed in a rather unhelpful way.&lt;/p&gt;

&lt;p&gt;Bonus problem: We lacked automated alerting for Sprite URLs, so we didn’t notice the outage as quickly as we should have. Once the incident was declared, we reissued the missing cert within 10 minutes.&lt;/p&gt;

&lt;p&gt;The proxy has since been fixed to prevent this from happening again, and we’re continually improving our alerting on the new Sprites surfaces.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Sprite creates failed due to oversized container image</title>
    <link rel="alternate" href="https://fly.io/infra-log/sprite-creates-failed/"/>
    <id>https://fly.io/infra-log/sprite-creates-failed/</id>
    <published>2026-10-21T00:00:00+00:00</published>
    <updated>2026-10-28T18:12:58+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;A routine update to the Sprites base image added enough content to exceed Machines&amp;rsquo; 8 GiB root filesystem limit. When workers tried to pull the new image, they ran out of space mid-extraction and the pull failed. Our orchestrator didn&amp;rsquo;t distinguish &amp;ldquo;this image will never fit&amp;rdquo; from a transient pull error, so it kept retrying. We mitigated the incident by rolling back to the previous image build.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>London BGP outage</title>
    <link rel="alternate" href="https://fly.io/infra-log/london-bgp-outage/"/>
    <id>https://fly.io/infra-log/london-bgp-outage/</id>
    <published>2026-10-21T00:00:00+00:00</published>
    <updated>2026-10-28T18:12:58+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;In the process of upgrading our edge servers in LHR, it turned out we decommissioned the old hosts before confirming the new ones were correctly advertising routes over BGP (hint: they were not). The result was a loss of inbound connectivity through LHR edges for roughly 15 minutes. We restored service by bringing the old edge servers back online, and BGP re-established normally once they were up. Our internal health check for BGP state should have caught this before the old servers were shut down, but it was passing on the new hosts even though BGP wasn&amp;rsquo;t actually working. We&amp;rsquo;re investigating why that happened and tightening that check so this doesn&amp;rsquo;t slip through the same crack later on.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Broken routing in GRU</title>
    <link rel="alternate" href="https://fly.io/infra-log/broken-routing-gru/"/>
    <id>https://fly.io/infra-log/broken-routing-gru/</id>
    <published>2026-10-14T00:00:00+00:00</published>
    <updated>2026-10-28T18:12:58+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;Routing in our GRU region was degraded for approximately two and a half hours. During this period, changes to app configuration and health probes did not always propagate consistently across the region. As a result, some requests to Machines in GRU returned &lt;code&gt;503&lt;/code&gt; errors, and some applications experienced inconsistent routing behavior.&lt;/p&gt;

&lt;p&gt;Other regions were unaffected, and all existing Machines continued running throughout the incident.&lt;/p&gt;
&lt;h3 id='root-cause' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#root-cause' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Root Cause&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;The incident was triggered by a rollout of a new replication metadata format for Corrosion, our distributed state-propagation system, to the GRU regional cluster. A bug in the new code prevented certain updates from replicating correctly between nodes, leaving routing information inconsistent across the cluster. As a result, &lt;code&gt;fly-proxy&lt;/code&gt; identified some backends as unhealthy or missing and returned &lt;code&gt;503&lt;/code&gt; responses.&lt;/p&gt;
&lt;h3 id='resolution' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#resolution' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Resolution&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;We rolled back the metadata format change, restored the cluster from a backup, and initiated a cluster-wide data synchronization. However, the cluster remained out of sync due to a separate configuration issue that prevented communication between Corrosion nodes.&lt;/p&gt;

&lt;p&gt;A recent configuration change had updated the IP address Corrosion uses to listen for peer connections, but the corresponding address registered in Consul, which nodes use to discover one another, had not been updated. This mismatch had gone unnoticed because nodes continued using previously cached peer addresses. Rebuilding the cluster from a backup exposed the issue when nodes attempted to rediscover their peers.&lt;/p&gt;

&lt;p&gt;We rolled back the configuration change, immediately restoring communication between nodes. The cluster&amp;rsquo;s state converged within a few minutes, and routing recovered.&lt;/p&gt;
&lt;h3 id='follow-up-actions' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#follow-up-actions' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Follow-up Actions&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Our existing testing did not catch this issue before it reached production. We are improving our testing and monitoring to help identify similar issues earlier and reduce the likelihood of recurrence. We are also continuing to test the new metadata format before proceeding with a further rollout.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Network readiness checks stalled worker hosts during reroute</title>
    <link rel="alternate" href="https://fly.io/infra-log/network-readiness-checks-stalled/"/>
    <id>https://fly.io/infra-log/network-readiness-checks-stalled/</id>
    <published>2026-10-09T00:00:00+00:00</published>
    <updated>2026-10-28T18:12:58+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;As we covered in a &lt;a href='https://fly.io/infra-log/gateways-broke-6pn/' title=''&gt;recent infra log&lt;/a&gt;, we have been working hard on improvements to our 6PN private networking this year. The goals for this work were to improve reliability and unlock some new tricks, one of which is 6PN rerouting.&lt;/p&gt;

&lt;p&gt;The gist is that, with the added flexibility in our internal routing, we can now reroute 6PN traffic to avoid network degradation. When our monitoring detects loss between two datacenters, we route 6PN traffic through an intermediary datacenter with a healthy connection to both ends, if one exists. This is something we have been doing for a couple of months already, we just haven’t talked about it much. The good news is that it has made our platform more resilient to upstream networking issues, and private networking between regions has been a lot more reliable since.&lt;/p&gt;

&lt;p&gt;In this incident, our network monitoring detected packet loss into our Mumbai (BOM) region from a range of other datacenters. In response, our reroute system kicked in and selected datacenters in Singapore (SIN) as relays for much of the traffic. A few hundred hosts worldwide switched their Mumbai-bound traffic paths to SIN at roughly the same time. The reroutes themselves were fine, but every source host first probed the candidate relay workers with a readiness check to confirm they could forward traffic. That check did a full dump of all network interfaces and addresses via netlink on the target host, which serialized behind a global kernel lock. Each dump took about 500ms on a busy host, so when 100+ source hosts hit the same relay worker simultaneously, we instantly queued up over 50 seconds of serialized kernel work. This, unsurprisingly, caused broad issues, and the impacted hosts went dark while connections stalled. A similar pattern hit workers in LAX about six hours later.&lt;/p&gt;

&lt;p&gt;The fix was straightforward: we single-flighted the readiness check so concurrent requests share a result, and replaced the full netlink interface/address dumps with a targeted lookup filtered to the one interface we actually care about. Response time for the check dropped from 511ms to 29ms, and 100 concurrent requests no longer duplicate the check.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Our gateways took out 6PN for a bit</title>
    <link rel="alternate" href="https://fly.io/infra-log/gateways-broke-6pn/"/>
    <id>https://fly.io/infra-log/gateways-broke-6pn/</id>
    <published>2026-10-08T00:00:00+00:00</published>
    <updated>2026-10-28T18:12:58+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;We broke private networking (6PN) for a certain subset of machines. The root cause was a collision between how we assign internal IP addresses to gateway nodes and how we route 6PN traffic to machines.&lt;/p&gt;

&lt;p&gt;To recap a bit how 6PN works: when a machine is first created, it is assigned a 6PN address in the form &lt;code&gt;fdaa:netid:netid:hostid:hostid:machid:machid:2&lt;/code&gt;. The &lt;code&gt;hostid:hostid&lt;/code&gt; octets are derived based on the physical host where the machine is located at the time of creation (or on migration, before ~May 2025). It is also important for routing purposes: even after we made 6PN addresses stable back in 2025, for machines that have never been migrated, the custom eBPF program we use for 6PN routing still by default makes routing decisions solely based on &lt;code&gt;hostid&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This year, we have been working on improvements to our 6PN setup, and part of this plan is to remove more of our custom routing implementation, which has been hard to debug and make any changes to at all, and lean on standard Linux routing tables. This also gives us much higher flexibility in defining how packets are routed internally in our network, unblocking a lot of reliability and performance work, which you should hear about in separate updates soon. An important part of this work is to decouple host IDs entirely from 6PN routing: machines&amp;rsquo; routes are now determined entirely by &lt;em&gt;where they are currently located&lt;/em&gt;, not how their addresses happen to be assigned.&lt;/p&gt;

&lt;p&gt;One headache to deal with in this new reality is gateways. There are two types of gateway peers: static and interactive. Static peers are those you would get from &lt;code&gt;fly wireguard create&lt;/code&gt;, which you can import into a standard Wireguard client to connect into your organization&amp;rsquo;s 6PN network. Interactive peers are those used by &lt;code&gt;flyctl&lt;/code&gt; for transient operations such as &lt;code&gt;fly ssh console&lt;/code&gt;. The difference is that static peers have a persistent identity in our platform&amp;rsquo;s routing state, while interactive peers do not. They can only be routed based on looking at the host ID, which, in this case, will be the gateway&amp;rsquo;s host ID.&lt;/p&gt;

&lt;p&gt;There are no other flags in the address for us to distinguish gateway peers from machines, so the easiest solution here was to just route the entire gateway&amp;rsquo;s &lt;code&gt;/80&lt;/code&gt; prefix (per &lt;code&gt;netid&lt;/code&gt;) to the gateway host by adding such a rule in Linux&amp;rsquo;s routing table. This worked perfectly fine for our current gateways! However, one separate line of work was to make our edges act as gateways for non-static peers. This means that every edge had to be treated like a gateway in our 6PN routing logic, and this was the direct cause of this incident:&lt;/p&gt;

&lt;p&gt;When a physical worker host is decommissioned, its internal IP address is released. This also determines its host ID. A newly provisioned host can either pick up a brand new IP / host ID or one from a decommissioned host. Ever since 6PN addresses became stable, this fact means that it is possible for machines to retain host IDs that end up being re-assigned to a non-worker host. This is exactly what happened when we started treating edges as gateways: they get &lt;code&gt;/80&lt;/code&gt; routes, which ended up overriding the code that handled per-machine routing (it depends on machines that have not yet been accessed locally &lt;em&gt;not having a route&lt;/em&gt;). Remember, machines that have those edges&amp;rsquo; host IDs (by retaining 6PN from the original worker host)  are most certainly NOT located on the edge (they have to be on workers by definition!).&lt;/p&gt;

&lt;p&gt;We mitigated the first occurrence within about 10 minutes by deleting the bad &lt;code&gt;/80&lt;/code&gt; routes from the routing table on all affected hosts and rolling back changes to &lt;code&gt;s6pnforwarderd&lt;/code&gt;, the daemon that manages 6PN routes. We then shipped a fix by only ever adding interactive peers as &lt;code&gt;/128&lt;/code&gt;. This, unfortunately, did not fix the root cause of the incident: it is not &lt;em&gt;just&lt;/em&gt; the &lt;code&gt;/80&lt;/code&gt; routes that are the problem but the priority between the gateway logic and worker logic. This caused a brief recurrence of this issue the following day. We then fixed the root cause at least for the moment by always looking for any machines that may bear a host&amp;rsquo;s ID first, regardless of whether the host is known to be a worker or not.&lt;/p&gt;

&lt;p&gt;This incident exposed a lot of issues with our 6PN setup. First and foremost, not everybody even &lt;em&gt;realized&lt;/em&gt; as a first reaction that a worker&amp;rsquo;s host ID may be reused by a non-worker host. This is not surprising if one knows how host ID assignment worked internally, but is really not well-documented. Our gateways have not been touched for a long time, and this is the only reason the issue did not surface until we started using edges as gateways. We have provisioned a lot of new edges recently to grow our capacity and for better connectivity, and that is why they ended up reusing decommissioned workers&amp;rsquo; host IDs.&lt;/p&gt;

&lt;p&gt;A deeper issue is that we need better alerting and automated testing for 6PN routing. All of our existing checks use machines sitting on their original host, which means they&amp;rsquo;d never catch a conflict caused by host ID reuse, or really any issue at all related to the stable 6PN address implementation. It is, in our opinion, quite amazing that we only got bit by this right now, which is definitely &lt;strong class='font-semibold text-navy-950'&gt;&lt;em&gt;not&lt;/em&gt;&lt;/strong&gt; a good thing. Many, many problems could have sneaked under our radar because they didn&amp;rsquo;t cause an alert or a more widespread outage like this. As part of the 6PN reliability improvement project, our plan is to, for now, at least assign non-local 6PN addresses to machines we use to test and alert on 6PN issues. The medium-to-long term goal is that host IDs in 6PN addresses should carry exactly no weight in deciding how they&amp;rsquo;re routed. This is already (sort of) true today for machines, but not true for gateway peers. It needs to be true for 6PN in general, lest we leave the door open for this type of incident to recur. The current state is a hack to keep existing gateway peers working (since we can&amp;rsquo;t force all existing peers and machines to a new 6PN address), but we really need to transition to a better model across the board for new peers and machines.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Sprites API outage during database migration</title>
    <link rel="alternate" href="https://fly.io/infra-log/sprites-api-outage/"/>
    <id>https://fly.io/infra-log/sprites-api-outage/</id>
    <published>2026-10-08T00:00:00+00:00</published>
    <updated>2026-10-08T22:48:19+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;Sprites API operations, including creating, listing, and restarting sprites, were unreliable for about 7 hours and 43 minutes. This happened off the back of a migration to PostgreSQL, which activated background database processes for thousands of previously inactive organizations. A metrics collector running every 15 seconds kept related cluster connections active, while a replication bug caused unnecessary data transfers between nodes. Together, these produced enough disk and network load to disrupt the service.&lt;/p&gt;

&lt;p&gt;We tried several mitigations to reduce background work and improve recovery, but they did not stop the underlying problem. We also tested a different hypervisor while investigating a possible I/O issue, then reverted when performance got worse. The main fixes were to start database processes only when needed, disable the metrics collector, and correct the replication bug. We have since replaced the collector with one that reads local state without opening cluster connections, added monitoring for disk and network pressure, and removed the old database code after a move to PostgreSQL.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>WireGuard gateways couldn't reach recently created or moved Machines via 6PN</title>
    <link rel="alternate" href="https://fly.io/infra-log/wireguard-gateways-6pn/"/>
    <id>https://fly.io/infra-log/wireguard-gateways-6pn/</id>
    <published>2026-10-05T00:00:00+00:00</published>
    <updated>2026-10-08T22:48:19+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;&lt;code&gt;s6pnforwarderd&lt;/code&gt;, the daemon that sets up 6PN (our internal IPv6 private networking) forwarding on WireGuard gateways, was crash-looping on some gateway hosts. 6PN on our gateways is configured slightly differently than on workers, and some files expected by the daemon were not found after a recent rollout. We are in the middle of reworking our 6PN setup for better performance and reliability, hence the rollout &amp;ndash; gateways were not touched for a long time before this, and some of their peculiarities were forgotten.&lt;/p&gt;

&lt;p&gt;This incident was not helped by the fact that we also recently switched our alerting setup, which somehow missed a number of checks on gateways during the conversion. This means that we didn&amp;rsquo;t notice this for a while (because all existing machines&amp;rsquo; 6PN kept working on gateways). Once we noticed though, the fix was quick and we just had to place some files at the expected locations and restart &lt;code&gt;s6pnforwarderd&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;We have since carried out an audit for our new alerting setup to catch any missing alerts and reinstate them. For gateways, our plan is to rework their setup to match what&amp;rsquo;s on workers, so that we do not need to always keep in mind that files are at slightly different locations based on host type. All hosts that run the 6PN stack should share more or less a similar setup. We are also planning to merge some functionalities of gateways with Anycast edges, instead of keeping them on an obscure host type that we seldom work on or notice.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Sprite create failures due to network instability</title>
    <link rel="alternate" href="https://fly.io/infra-log/sprite-create-failures/"/>
    <id>https://fly.io/infra-log/sprite-create-failures/</id>
    <published>2026-10-01T00:00:00+00:00</published>
    <updated>2026-10-08T22:48:19+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;Sprite SQLite databases became unreachable from worker nodes in multiple regions including SIN, DFW, LHR, IAD, FRA, and ORD. The errors came in two spikes: the first lasted roughly six minutes, the second about two minutes. During these spikes, existing Sprites continued to run, but impacted organizations could not create new ones. Both Spikes correlated with instability across our internal network.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Stuck DNS broke some Depot builds</title>
    <link rel="alternate" href="https://fly.io/infra-log/stuck-dns-depot-builds/"/>
    <id>https://fly.io/infra-log/stuck-dns-depot-builds/</id>
    <published>2026-09-28T00:00:00+00:00</published>
    <updated>2026-10-08T22:48:19+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;A number of build attempts failed over a roughly 24-hour period, with affected builds hanging for about five minutes before timing out. The failures were concentrated on two ingress machines in SYD and JNB that had stopped properly routing connections to builders.&lt;/p&gt;

&lt;p&gt;The underlying cause was a bug in &lt;code&gt;corro-dns&lt;/code&gt;, the internal DNS resolver we use for service discovery. &lt;code&gt;corro-dns&lt;/code&gt; deduplicates concurrent lookups for the same record: if a refresh is already in flight, new callers subscribe to that same refresh rather than starting another. If the caller&amp;rsquo;s patience runs out (about one second), it falls back to the cached answer. The problem is that when the background refresh itself hangs, it never completes, and every subsequent lookup for that record subscribes to the same stuck refresh, waits a second, and gets the same stale cached answer, forever. For most records the stale data was still close enough to correct that nothing visibly broke. But &lt;code&gt;_instances.internal&lt;/code&gt;, a heavy query used for builder discovery, went stale in a way that left the affected ingress machines unable to find running builders. The machines kept accepting connections but couldn&amp;rsquo;t route them, so clients saw TLS handshakes fail or time out.&lt;/p&gt;

&lt;p&gt;Once we became aware of the issue we cordoned the two affected ingress machines, which immediately restored builds for affected users. We then restarted &lt;code&gt;corro-dns&lt;/code&gt; on those hosts, confirmed the DNS results were fresh again, uncordoned the machines, and verified traffic was flowing.&lt;/p&gt;

&lt;p&gt;The short-term answer is that we&amp;rsquo;ll add a timeout to the refresh path in &lt;code&gt;corro-dns&lt;/code&gt; so a single hung query can&amp;rsquo;t permanently poison the cache for that record. Also, equally importantly, queries shouldn&amp;rsquo;t hang. We&amp;rsquo;re also planning a bigger overhaul of our internal DNS setup &amp;ndash; stay tuned!&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Sprites API partial outage from SQLite database evictions</title>
    <link rel="alternate" href="https://fly.io/infra-log/sprites-api-sqlite-evictions/"/>
    <id>https://fly.io/infra-log/sprites-api-sqlite-evictions/</id>
    <published>2026-09-16T00:00:00+00:00</published>
    <updated>2026-09-18T09:42:53+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;Sprites API stores per-org state in lightweight SQLite databases, each managed by a local (to the organization) control plane node. The nodes periodically write per-database heartbeats to object storage to prove they&amp;rsquo;re alive. If a node misses a heartbeat, it assumes something is wrong and evicts all of its databases, which then attempt to restore from their Litestream replicas on other nearby nodes.&lt;/p&gt;

&lt;p&gt;For about 30 minutes, multiple control plane nodes started failing their heartbeat writes. The failures hit nodes across several regions but were primarily concentrated in Europe. Each failed heartbeat triggered a bulk eviction, displacing hundreds of databases to other failing nodes. Databases continued to hop between nodes until heartbeats resumed.&lt;/p&gt;

&lt;p&gt;We&amp;rsquo;re still not sure exactly what caused the heartbeat writes to start failing. Our leading hypothesis is intermittent connectivity loss between our nodes and Tigris (our object storage provider), amplified by a bug in the retry logic that made subsequent attempts always fail.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Sidekiq periodic scheduler not running after Redis timeout</title>
    <link rel="alternate" href="https://fly.io/infra-log/sidekiq-scheduler-redis/"/>
    <id>https://fly.io/infra-log/sidekiq-scheduler-redis/</id>
    <published>2026-09-11T00:00:00+00:00</published>
    <updated>2026-09-18T09:42:53+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;Sidekiq uses a leader election to decide which worker process enqueues periodic (aka cron) jobs. The newly elected leader hit a transient network issue communicating with Redis at exactly the wrong moment: the process still became the elected scheduler, but it never completed initialization of its periodic job manager. That left scheduled background work not being enqueued for roughly five hours. The missed jobs included billing usage aggregation, certificate renewal checks, and rechecks for newly requested certificates that had not passed their initial domain validation.&lt;/p&gt;

&lt;p&gt;Once we detected the problem, we restarted the wedged scheduler process to restore periodic job execution, and backfilled the missed billing usage.&lt;/p&gt;

&lt;p&gt;We’re adding direct monitoring for missed periodic jobs so it doesn’t take us five hours to notice, and improving the reliability of the network between the Sidekiq workers and Redis.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Fly-proxy HTTP/2 routing broke due to missing Consul service rows</title>
    <link rel="alternate" href="https://fly.io/infra-log/http2-routing-consul/"/>
    <id>https://fly.io/infra-log/http2-routing-consul/</id>
    <published>2026-09-09T00:00:00+00:00</published>
    <updated>2026-09-09T20:25:09+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;A manual cleanup/config change to our Consul service registration data briefly removed the HTTP/2 routing entries for fly-proxy on a subset of hosts. During that window, some HTTP/2-based traffic to Machines on those hosts could fail. We mitigated the issue by restarting the service-registration sync agent on affected hosts/regions to republish the missing entries, restoring normal routing.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Stuck background job queue blocked Sprite deletions</title>
    <link rel="alternate" href="https://fly.io/infra-log/job-queue-blocked/"/>
    <id>https://fly.io/infra-log/job-queue-blocked/</id>
    <published>2026-09-08T00:00:00+00:00</published>
    <updated>2026-09-18T09:42:53+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;An Oban background job queue used by Sprites got stuck when repeated attempts to place it on a Sprites API Machine failed. This caused Sprite deletions and some other background operations to fail. The queue recovered automatically about an hour later, and Sprite deletion requests began succeeding again.&lt;/p&gt;

&lt;p&gt;Investigation traced the placement failures to a Sprites API Machine with a corrupted ext4 volume. The Machine had been migrated between hosts several days earlier, which led us to investigate the migration path.&lt;/p&gt;

&lt;p&gt;When we migrate a volume, we use Linux&amp;rsquo;s &lt;code&gt;dm-clone&lt;/code&gt; to allow the destination Machine start before the entire volume has been copied: reads for uncopied blocks come from the source host, while writes and already-copied blocks use the destination disk. After the copy completes, the running Machine continues using the clone device until its next restart.&lt;/p&gt;

&lt;p&gt;Although the clone and its underlying destination ultimately access the same storage, Linux treats them as separate block devices and maintains an independent page cache for each. A nightly volume check read filesystem metadata directly from the destination device and cached it there. The running Machine continued updating that metadata through the clone, but those writes did not invalidate the destination device’s cached copy. When the Machine was eventually restarted, it read a mixture of stale metadata and current disk contents, corrupting the filesystem.&lt;/p&gt;

&lt;p&gt;We now ensure the kernel has invalidated the underlying device&amp;rsquo;s cache before allowing a Machine to start. The volume check and other maintenance commands have also been updated to read through the clone device when it exists, preventing the incoherent cache from being created in the first place.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>WireGuard gateways broke after schema migration mismatch</title>
    <link rel="alternate" href="https://fly.io/infra-log/wireguard-schema-mismatch/"/>
    <id>https://fly.io/infra-log/wireguard-schema-mismatch/</id>
    <published>2026-09-04T00:00:00+00:00</published>
    <updated>2026-09-07T14:41:08+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;We ran a routine deploy of &lt;code&gt;wggwd&lt;/code&gt;, the service that sets up customer WireGuard peers, which shouldn&amp;rsquo;t have impacted anything as the latest code was already deployed (or so we thought). Soon after the deploy we noticed new peer additions failing with a SQL error (wggwd runs a sqlite database on each server to cache peers).&lt;/p&gt;

&lt;p&gt;This was tracked down to a new feature whose SQL migration had been merged but not deployed; wggwd was erroring on a &lt;code&gt;select *&lt;/code&gt; that picked up more columns than the code expected. However, we were confused by the error, since it was not possible for the wggwd service to &lt;em&gt;both&lt;/em&gt; have restarted to run migrations, and not restarted as to fail on the new schema.
It turns out this was caused by our alerting, that runs a &lt;code&gt;wggwd check&lt;/code&gt; command on an interval; the check command was incorrectly set up to run migrations when opening the database connection.&lt;/p&gt;

&lt;p&gt;We closed out this incident by restarting wggwd everywhere to pick up the new code, fixing the check command so it doesn&amp;rsquo;t run migrations, and ensuring deployments actually restart the server process.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Metrics backlog</title>
    <link rel="alternate" href="https://fly.io/infra-log/metrics-backlog/"/>
    <id>https://fly.io/infra-log/metrics-backlog/</id>
    <published>2026-09-02T00:00:00+00:00</published>
    <updated>2026-09-07T14:41:08+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;A surge of hot &amp;amp; fresh time-series blew out our storage/query cardinality and caused metrics to appear delayed by 30-60 minutes. We spun up some short term metrics capacity to absorb the load and caught up on the backlog without dropping anything along the way. Once caught up, the metrics cluster has been ticking away happily, albeit at its presumably anomalous new volume. (We&amp;rsquo;ll get to that soon.)&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Development node advertised Fly public DNS anycast IPs and broke it</title>
    <link rel="alternate" href="https://fly.io/infra-log/dns-anycast-break/"/>
    <id>https://fly.io/infra-log/dns-anycast-break/</id>
    <published>2026-08-28T00:00:00+00:00</published>
    <updated>2026-09-07T14:41:08+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;An internal development server was misconfigured to advertise our public DNS&amp;rsquo;s IP address, but the node&amp;rsquo;s DNS service had been failing. As a result, some recursive resolvers routed queries to that location and experienced brief DNS resolution errors. We resolved this by removing that server from the anycast/DNS serving range so it stopped receiving DNS traffic.&lt;/p&gt;

&lt;p&gt;Of course, the most obvious answer here going forward is that this kind of misconfiguration should not happen again, and we need to have more aggressive alerts about misconfigured DNS. Another route of improvement is to make our two DNS anycast IPs &lt;em&gt;actually&lt;/em&gt; redundant to each other by announcing them from different subsets of PoPs.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Some API endpoints briefly rejected flyctl requests</title>
    <link rel="alternate" href="https://fly.io/infra-log/api-rejected-flyctl/"/>
    <id>https://fly.io/infra-log/api-rejected-flyctl/</id>
    <published>2026-08-27T00:00:00+00:00</published>
    <updated>2026-09-07T14:41:08+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;We deployed a change to one of our API backends to reject authentication using legacy OAuth &amp;ldquo;f01&lt;em&gt;&amp;rdquo; tokens. Due to a bug, this also ended up rejecting the token &amp;ldquo;bundles&amp;rdquo; used by flyctl - these bundles include one Macaroon token per org, as well as an user-specific OAuth token which uses the same &amp;ldquo;f01&lt;/em&gt;&amp;rdquo; prefix as the legacy tokens.&lt;/p&gt;

&lt;p&gt;We rolled out an update to the token parsing/rejection logic to allow mixed-token bundles (while still rejecting legacy tokens), and &lt;code&gt;flyctl&lt;/code&gt; behavior returned to normal; we also added a code test to help catch similar issues in the future.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>MPGv1 clusters in ORD-0 failed to restart</title>
    <link rel="alternate" href="https://fly.io/infra-log/mpgv1-restart-ord/"/>
    <id>https://fly.io/infra-log/mpgv1-restart-ord/</id>
    <published>2026-08-27T00:00:00+00:00</published>
    <updated>2026-09-07T14:41:08+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;While moving the underlying orchestration for our legacy Managed Postgres v1 service in ORD, some of the control-plane Machines didn’t auto-start (and briefly re-stopped), which prevented a number of ORD-0 Postgres clusters from restarting cleanly. Most clusters recovered quickly once the orchestration layer caught up, but a smaller set remained degraded because replica Machines could not be recreated, including failures returning “insufficient resources to create new machine with existing volume.” We resolved the incident by letting the control plane recover and then manually rescheduling/repairing the remaining affected replicas until all health checks were passing again.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Petsem primary crash loop from corrupted volume</title>
    <link rel="alternate" href="https://fly.io/infra-log/petsem-crashloop-volume/"/>
    <id>https://fly.io/infra-log/petsem-crashloop-volume/</id>
    <published>2026-08-25T00:00:00+00:00</published>
    <updated>2026-09-07T14:41:08+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;The primary node of Petsem (our secrets database) suffered from disk corruption during a routine deployment.
Petsem can run in a degraded state when the primary is unavailable, so while creating new apps and updating secrets failed during the impact period, it was still possible to perform read actions, such as deploying new machines.&lt;/p&gt;

&lt;p&gt;We restored service by promoting an up-to-date replica to become the new primary, repointing the other replicas to follow it, and then provisioning a new replica to return the cluster to full redundancy.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Stuck nftables reload blocked flyd startup causing API errors</title>
    <link rel="alternate" href="https://fly.io/infra-log/nftables-reload-flyd/"/>
    <id>https://fly.io/infra-log/nftables-reload-flyd/</id>
    <published>2026-08-24T00:00:00+00:00</published>
    <updated>2026-09-07T14:41:08+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;While deploying an &lt;code&gt;nftables&lt;/code&gt; change to our worker hosts, a simultaneous &lt;code&gt;flyd&lt;/code&gt; (our orchestrator for Fly Machines) deployment happened which saw a number of hosts stuck trying to restart &lt;code&gt;flyd&lt;/code&gt;. Because of this, API requests targeted at machines on those hosts were temporarily failing.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;flyd&lt;/code&gt; was stuck restarting because its &lt;code&gt;systemd&lt;/code&gt; service was waiting for &lt;code&gt;nftables&lt;/code&gt; to reload, due to &lt;code&gt;Requires=nftables.service&lt;/code&gt; and &lt;code&gt;After=nftables.service&lt;/code&gt; clauses. These clauses are necessary to ensure that we do not accidentally expose customer VMs to the Internet in case of &lt;code&gt;nftables&lt;/code&gt; issues. However, in this case, &lt;code&gt;nftables&lt;/code&gt; was taking an abnormally long time to reload, which in turn blocked &lt;code&gt;flyd&lt;/code&gt; from restarting in the interim (note: this is due to &lt;code&gt;systemd&lt;/code&gt;&amp;lsquo;s service startup ordering; in this case, it is not &lt;em&gt;strictly necessary&lt;/em&gt; to queue &lt;code&gt;flyd&lt;/code&gt; like this since the &lt;code&gt;nftables&lt;/code&gt; service was never being fully restarted, only reloaded, but it is how &lt;code&gt;systemd&lt;/code&gt; works).&lt;/p&gt;

&lt;p&gt;We traced the slow &lt;code&gt;nftables&lt;/code&gt; reload down to a suboptimal save/restore script we use for the purpose of machines&amp;rsquo; network policies &amp;ndash; they are implemented as &lt;code&gt;nftables&lt;/code&gt; rules and must be restored on reload. The script to reload these rules was looping through every single machine on a host calling &lt;code&gt;nft list chain&lt;/code&gt; repeatedly, which added up to a long time on the more busy hosts. It would not have been a problem if &lt;code&gt;flyd&lt;/code&gt; wasn&amp;rsquo;t being deployed at the same time, either. This incident eventually resolved itself once &lt;code&gt;nftables&lt;/code&gt; finished reloading, but we&amp;rsquo;ll optimize how the reload script works so that this does not happen when we need to reload &lt;code&gt;nftables&lt;/code&gt; again. We would like to also optimize the way we use &lt;code&gt;nftables&lt;/code&gt; for network policies to reduce the sizes of &lt;code&gt;nftables&lt;/code&gt; on worker hosts.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Egress IP issues in SJC</title>
    <link rel="alternate" href="https://fly.io/infra-log/egress-ip-sjc/"/>
    <id>https://fly.io/infra-log/egress-ip-sjc/</id>
    <published>2026-08-19T00:00:00+00:00</published>
    <updated>2026-08-19T18:18:52+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;We saw intermittent outbound connectivity failures for traffic using static egress IP addresses in SJC. The issue turned out to be caused by an unhealthy WireGuard tunnel between a worker host and its egress gateway. This is, in fact, a known bug internally: sometimes our WireGuard tunnels just lose connectivity to one of the peers. Usually this requires a reset of the WireGuard tunnel (fun fact: if you do it wrong, resetting the tunnel might also deadlock the rest of the kernel&amp;rsquo;s network stack), which we usually will do when we receive an alert for this. Unfortunately, on this day, that specific alert was caught in the middle of another alert storm, and was not noticed and handled for a significant amount of time. We were able to resolve this quickly once we realized this fact, but this incident did tell us a few things we need to work on:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The alert and incident handling process. This is something we now have an entire team for, and we are hopeful that ignoring alerts inadvertently during an alert storm will be a thing of the past soon;
&lt;/li&gt;&lt;li&gt;Reliability of our internal network mesh (WireGuard) itself. This bug is already fixed upstream, so any new hosts we have provisioned recently will not see the same issue. For existing hosts, kernel upgrades will be applied as they are gradually rebooted. Of course, this is a somewhat slow process and unless we decided to reboot every existing host at once, we&amp;rsquo;ll likely continue to see these issues from time to time. Fortunately, we have been working on the ability to reroute traffic over the WireGuard mesh when this kind of connectivity issues happen, including but not limited to this WireGuard bug. Once that work is completed, hopefully these occasional issues will cause much less disruption from users&amp;rsquo; perspective.
&lt;/li&gt;&lt;/ol&gt;
</content>
  </entry>
  <entry>
    <title>Global Corrosion lag caused Machines API app-not-found errors</title>
    <link rel="alternate" href="https://fly.io/infra-log/machines-api-errors/"/>
    <id>https://fly.io/infra-log/machines-api-errors/</id>
    <published>2026-08-17T00:00:00+00:00</published>
    <updated>2026-08-17T14:50:41+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;A subset of very large service-definition updates from machines with large numbers of services could not be applied to our global service-discovery database (Corrosion) within the configured timeout on some nodes. These updates were repeatedly retried, slowing down or blocking the application of other updates. Since the Machines API (via Flaps) reads app metadata from global Corrosion, some newly created apps and recent updates were not visible everywhere, resulting in elevated app-not-found errors.&lt;/p&gt;

&lt;p&gt;We mitigated the issue by adjusting the timeout and Corrosion’s retry behavior so that failed updates do not retry indefinitely and block other updates. We also rolled out a fix to chunk updates when a machine has a large number of services, preventing such updates from causing similar propagation issues in the future.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Certificate issuance delays from Sidekiq overload</title>
    <link rel="alternate" href="https://fly.io/infra-log/sidekiq-cert-delays/"/>
    <id>https://fly.io/infra-log/sidekiq-cert-delays/</id>
    <published>2026-08-11T00:00:00+00:00</published>
    <updated>2026-08-17T14:50:41+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;A surge of certificate-check and renewal work overwhelmed our background job system, causing significant delays issuing and renewing TLS certificates. This was caused by a few reasons, but the main one being an internal DNS resolver dedicated to the certificate renewal path: it is configured with no caching and aggressively recurses to upstream whenever possible in order to minimize delays in observing DNS updates for customers&amp;rsquo; domains. That, however, was initially configured at a time where we were operating at a much smaller scale than we are today, and on this specific day, a spike of DNS renewal tasks combined with possibly slow lookups for some domains completely paralyzed the TLS certificate issuance queue. In the process of investigating this issue, we also discovered that the TLS certificate renewal task has been trying to renew certificates for deleted apps (with no success, of course), which did not help with the excessive queuing delays seen on this day.&lt;/p&gt;

&lt;p&gt;We have put in several mitigations for this. Firstly, the DNS resolver used for TLS certificate renewal is now allowed to do some minimal caching to act as a buffer against temporary spikes in TLS renewal tasks and domain resolution failures. This does mean that updated DNS configuration could take a little while to propagate to our certificate issuer, but since this is on the order of seconds, it should not be noticeable under normal operation, and it provided enough protection for us to resolve the Sidekiq queuing issue. We also fixed several issues in our codebase that did not filter out deleted apps correctly, so that we are less likely to experience these spikes in the first place.&lt;/p&gt;

&lt;p&gt;This is not the first time we&amp;rsquo;ve had Sidekiq-related incidents this year. We have plans to improve reliability around certificate issuance &amp;ndash; our earlier work to move certificates out of Vault due to numerous incidents being an example &amp;ndash; and another part of it is to decouple our certificate issuing pipeline from the rest of our API implementation, a Ruby on Rails app. The hope is that in this process we will make it both more scalable and easier to understand and debug.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>BGP misconfiguration dropped ingress traffic</title>
    <link rel="alternate" href="https://fly.io/infra-log/bgp-ingress-drop/"/>
    <id>https://fly.io/infra-log/bgp-ingress-drop/</id>
    <published>2026-08-11T00:00:00+00:00</published>
    <updated>2026-08-17T14:50:41+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;A BGP misconfiguration while provisioning new edge capacity caused most traffic from Europe (and some US) endpoints to be dropped. The misconfiguration has been fixed and we have implemented safeguards against this kind of issue in the future.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>IAD MPGv2 Patroni etcd lease-expiration storm</title>
    <link rel="alternate" href="https://fly.io/infra-log/patroni-etcd-storm/"/>
    <id>https://fly.io/infra-log/patroni-etcd-storm/</id>
    <published>2026-08-10T00:00:00+00:00</published>
    <updated>2026-08-11T04:46:17+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;MPG v2 in the IAD region saw a burst of Patroni health-check failures and timeouts when the region&amp;rsquo;s shared Patroni etcd cluster became overloaded. Existing databases were unaffected because Patroni entered failsafe mode, but new clusters could not be created. Investigation pointed to an intrinsic etcd problem: adding a new user bumps etcd&amp;rsquo;s Auth Revision, invalidating existing JWT tokens and forcing clients to reconnect. On reconnect, a thundering herd of clients overloaded etcd and brought it down. We mitigated by scaling up the Patroni etcd machines in IAD and lowering the bcrypt cost, and we&amp;rsquo;re exploring longer-term fixes.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>tkdb primary host outage caused token validation failures</title>
    <link rel="alternate" href="https://fly.io/infra-log/tkdb-primary-outage/"/>
    <id>https://fly.io/infra-log/tkdb-primary-outage/</id>
    <published>2026-08-04T00:00:00+00:00</published>
    <updated>2026-08-11T04:46:17+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;The host running the primary node of &lt;code&gt;tkdb&lt;/code&gt;, our token validation service, went offline due to maintenance and did not manage to reboot successfully due to a separate hardware issue. This caused token validation to time out, which in turn led to widespread failures in API operations that depend on authentication (Machines API, &lt;code&gt;flyctl&lt;/code&gt;, etc.) and elevated 500s from platform services. API and dashboard functionality recovered over a few minutes once we restored the host to full service.&lt;/p&gt;

&lt;p&gt;The root cause of this incident is, of course, the host failure, but a huge contributing factor is that &lt;code&gt;tkdb&lt;/code&gt; is single-primary, and, unlike &lt;code&gt;petsem&lt;/code&gt; (which powers app secrets and TLS certs), read replicas of &lt;code&gt;tkdb&lt;/code&gt; do not function if the primary node is entirely offline. This is because, as a token validation service, the read replicas must be able to receive notifications about revocation to uphold security guarantees. We are currently looking into ways we can relax this requirement, keeping replicas up during primary outages, while avoiding impact on security properties provided by tokens.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Missing app logs because we forgot a `systemctl restart`</title>
    <link rel="alternate" href="https://fly.io/infra-log/missing-app-logs/"/>
    <id>https://fly.io/infra-log/missing-app-logs/</id>
    <published>2026-08-04T00:00:00+00:00</published>
    <updated>2026-08-11T04:46:17+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;Some customers reported that their app logs appeared to be missing. Initially we suspected that this was related to log ingestion (we use &lt;code&gt;vector&lt;/code&gt; on each physical host), but further investigation revealed that&amp;hellip; we forgot to restart the query service after adding a new storage node to the logging cluster, so it only queried the previous set of nodes. Because each app’s log stream is written to one storage node, apps whose logs ended up stored on the new node would return empty or incomplete results. Restarting the log query service restored normal log visibility.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Vault throws one last tantrum on its way out the door</title>
    <link rel="alternate" href="https://fly.io/infra-log/vault-tantrum/"/>
    <id>https://fly.io/infra-log/vault-tantrum/</id>
    <published>2026-07-24T00:00:00+00:00</published>
    <updated>2026-07-29T15:14:59+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;Recently we have been migrating our certificate storage away from Vault to our Petsem codebase, to reduce the number of distinct cluster-shaped things we need to think about. We reached the 100% mark on the rollout a few days before this incident, so this body of work was all but done. &lt;/p&gt;

&lt;p&gt;This is, of course, when Vault decided to break in some manner. Since we were still validating things and hadn&amp;rsquo;t rolled out our code sans feature flags yet, there remained calls to Vault in both our GraphQL API and the Fly Proxy, particularly as a fallback for unknown certificates. Though these weren&amp;rsquo;t in the hot path of healthy requests, Vault hanging caused some issues across the services, and caused TLS handshakes to fail for otherwise functional Fly Apps for some number of minutes.&lt;/p&gt;

&lt;p&gt;Unlike most writeups here, we don&amp;rsquo;t have (or, need) a firm root cause on exactly what broke and where. As soon as this kicked off we made the call to rip out the Vault functionality, since that was up next anyway. Without Vault in play, we got to blissfully ignore whatever went wrong with it. And we didn&amp;rsquo;t have much of a reason to dig into exactly which Vault/Petsem interaction in the proxy misbehaved, as the code that housed it was razed.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>SJC worker hosts locked up and rebooted</title>
    <link rel="alternate" href="https://fly.io/infra-log/worker-lockups-sjc/"/>
    <id>https://fly.io/infra-log/worker-lockups-sjc/</id>
    <published>2026-07-21T00:00:00+00:00</published>
    <updated>2026-07-29T15:14:59+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;While debugging network packet loss in SJC, a &lt;code&gt;tc qdisc&lt;/code&gt; (queuing discipline) configuration update was (unnecessarily) pushed out to all of SJC workers, which caused a number of them to lock up. We promptly started rebooting them and nursing them back to health.&lt;/p&gt;

&lt;p&gt;Honestly, we are still not sure why this happened in the first place. The &lt;code&gt;tc qdisc&lt;/code&gt; configuration was, after further validation, later &lt;em&gt;very carefully&lt;/em&gt; deployed to all of our hosts &lt;em&gt;in all regions&lt;/em&gt;, and none experienced a similar problem whatsoever. &lt;code&gt;qdisc&lt;/code&gt; is not supposed to just randomly lock up entire systems: at most, it should cause network issues that we can easily recover from. So, it seems likely that we must have hit some weird kernel or NIC driver bug, but so far we do not know exactly which one. One thing is for sure though: we&amp;rsquo;ll have to be much more careful about even the most unassuming changes related to NICs.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>A couple of deployment issues</title>
    <link rel="alternate" href="https://fly.io/infra-log/deployment-issues/"/>
    <id>https://fly.io/infra-log/deployment-issues/</id>
    <published>2026-07-16T00:00:00+00:00</published>
    <updated>2026-10-08T22:48:19+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;This day saw 2 separate instances of deployment-related issues centered around IAD happening around the same time.&lt;/p&gt;

&lt;p&gt;The first one was related to Depot builders, where &lt;code&gt;flyctl deploy&lt;/code&gt; would wait seemingly indefinitely for a Depot builder to become available. Initially, we believed that this was related to a single IAD host under immense I/O pressure, and was only affecting a subset of Depot builders with backing volumes on that host. Failed hosts with Depot volumes can sometimes prevent existing Depot builders from being reused until recreated via an explicit &lt;code&gt;flyctl --recreate-builder&lt;/code&gt; flag. However, after we resolved the host issue (which took a considerable amount of time), it became clear that the Depot issue was not limited to the one host, but was impacting builds for any IAD builders. This is when an internal incident was declared and a status page posted.&lt;/p&gt;

&lt;p&gt;It turns out that this was another case of capacity-related Depot issue. Because machines used for Depot builders are pretty large, when we&amp;rsquo;re under capacity constraints, Depot builders may not be able to successfully start with an existing volume, because that host has no spare capacity to run the builder machine. Usually, machines can auto-migrate on start when capacity error happens, but this is explicitly not enabled for machines and volumes as large as builders. Under these circumstances, no new Depot builder should land in IAD, but existing ones will continue to be attempted and fail to start. The fix here, before adding more capacity, was to purge existing Depot builders from IAD in order to have them recreated in other North American regions as needed.&lt;/p&gt;

&lt;p&gt;Separately, our Docker registry, which runs as a Fly App managed by us, started throwing a lot of 5xx errors, &lt;em&gt;while the Depot issue above was happening&lt;/em&gt;. This accounted for another bulk of deployment issues happening on this day, which unfortunately compounded with Depot-related issues above. Initially we thought the host DNS resolvers were acting up, since the registry app is throwing a lot of DNS-related errors. After ruling that out, we realized that even pinging the host (&lt;code&gt;fdaa::3&lt;/code&gt;, where the DNS resolver also lives) from these machines was showing latency up to 100ms &amp;ndash; definitely too much for basically loopback traffic! At first, we thought this is due to CPU starvation since the machines were all at 100% utilization. Scaling up &lt;em&gt;just&lt;/em&gt; the CPU did bring down the latency, but it was still at several 10s of milliseconds. Some further investigation made us realize that these machines are running a lot of traffic over their TAP interfaces, and we do know that our TAP interfaces, for one reason or another, starts to struggle a bit under higher utilization (&amp;gt; 1Gbps). Creating more machines for the registry to spread the traffic out immediately solved the issue.&lt;/p&gt;

&lt;p&gt;We came out of this incident with some plans for improvement:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;We should know about widespread deployment issues way sooner than we did on this day. We did not put up a statuspage because we assumed that the issue was related to a downed host, which already had its separate host issue, but it was in fact unrelated. We now have better monitoring for Depot issues related to machine and volume placement, and we will hopefully be paged much, much sooner should something like this happen again.
&lt;/li&gt;&lt;li&gt;Even when there is capacity pressure, ideally, Depot builders should still be available, just placed in different regions, for new &lt;em&gt;or&lt;/em&gt; existing users. We are working with Depot to add support for placing builders using &lt;a href='https://docs.fly.io/machines/guides-examples/machine-placement#geographic-groups-and-aliases' title=''&gt;geo region aliases&lt;/a&gt;, which allows much more flexible machine placement, and we are planning to integrate this with our API and &lt;code&gt;flyctl&lt;/code&gt; so that local capacity issues no longer automatically translate to much wider deployment failures.
&lt;/li&gt;&lt;li&gt;For registry, besides similar alerting, figuring out the performance bottleneck on our TAP interfaces is on our plan as well. Fixing this will improve experience across all machines, with the caveat that we&amp;rsquo;ll likely still place constraints to ensure fairness between machines on a host.
&lt;/li&gt;&lt;/ol&gt;
</content>
  </entry>
  <entry>
    <title>Power supply failure in ORD</title>
    <link rel="alternate" href="https://fly.io/infra-log/power-failure-ord/"/>
    <id>https://fly.io/infra-log/power-failure-ord/</id>
    <published>2026-07-14T00:00:00+00:00</published>
    <updated>2026-07-15T21:01:52+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;For redundancy, servers generally have two power supplies, connected to two independent power feeds: if one power supply fails, or one power feed goes down, the server can keep operating on the other power supply. During normal operation, the load is shared between both supplies.&lt;/p&gt;

&lt;p&gt;Our servers are, of course, no exception. However, when one power feed of an ORD datacenter went down, we found our servers&amp;rsquo; CPUs heavily throttled, to the point where they were unable to do any useful work. Later discussing with the provider, we found a misconfigured setting: the server would throttle the CPUs when one power supply failed or lost power.&lt;/p&gt;

&lt;p&gt;Now, this isn&amp;rsquo;t a &amp;ldquo;oh duh, it should obviously not be like that&amp;rdquo; situation: this is a safety feature designed to not overload one power feed. Single-power feed operation, in some cases, can just be a fallback for the server to shut down safely without the ability to support normal tasks, even though this is not true in our case &amp;ndash; we expect it to provide better uptime at full performance. The throttling setting makes sense as a safe default, and we should be explicitly opting out of the safety feature when we are sure single-power feed operation is safe &lt;em&gt;in our case&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;As a result of this incident, we are now working with our providers to ensure all of our servers&amp;rsquo; settings are configured best for how our power feeds are set up.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Certificate issuance outage</title>
    <link rel="alternate" href="https://fly.io/infra-log/certificate-issuance-outage/"/>
    <id>https://fly.io/infra-log/certificate-issuance-outage/</id>
    <published>2026-07-14T00:00:00+00:00</published>
    <updated>2026-07-15T21:01:52+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;We issue certificates using &lt;a href='https://letsencrypt.org/' title=''&gt;Let&amp;rsquo;s Encrypt&lt;/a&gt;, who had a &lt;a href='https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/6a3416f4f7a0500610c184c5' title=''&gt;networking hiccup&lt;/a&gt; when failing over datacenters for maintenance. This caused some requests to fail, making our issuance jobs bail out and re-queue themselves. Apart from a handful of lucky hostnames who squeaked through, certificate issuance was out of action for a few hours. We renew certificates well before they expire, so no existing traffic was affected. Any apps that were in the process of setting up their hostname would have seen a delay in getting their first certificate. Once the upstream incident was resolved, all queued issuances and renewals were processed.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Runaway cleanup job filled Redis</title>
    <link rel="alternate" href="https://fly.io/infra-log/redis-cleanup-job/"/>
    <id>https://fly.io/infra-log/redis-cleanup-job/</id>
    <published>2026-07-14T00:00:00+00:00</published>
    <updated>2026-07-15T21:01:52+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;Our GraphQL API runs its background jobs on Sidekiq. When an app is deleted, we enqueue a cleanup job to tear down any associated resources, and that job in turn enqueues a job for every Machine the app ever had - even ones that had already been deleted.&lt;/p&gt;

&lt;p&gt;One app had a very large Machine history. Its cleanup job loaded that entire history into memory at once to enqueue the per-Machine jobs, which pushed the worker&amp;rsquo;s memory high enough that the supervisor recycled it - killing the process and restarting the job from scratch on another worker. The loop repeated and quickly piled up tens of millions of per-Machine jobs. That filled the disk on the Redis instance backing Sidekiq; once Redis could no longer write its append-only file, job processing stalled and customers saw API errors for around 40 minutes. Already-running Machines were unaffected and kept serving traffic.&lt;/p&gt;

&lt;p&gt;We mitigated by extending the Redis volume and restarting Redis, and shipped a change to make the cleanup job safer - it now iterates in smaller batches without pulling everything into memory and skips already-deleted Machines. Once Redis recovered, the backlog drained and error rates returned to normal.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>VictoriaMetrics ingestion delays and backlog</title>
    <link rel="alternate" href="https://fly.io/infra-log/victoriametrics-ingestion-delays/"/>
    <id>https://fly.io/infra-log/victoriametrics-ingestion-delays/</id>
    <published>2026-07-07T00:00:00+00:00</published>
    <updated>2026-07-15T21:01:52+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;We had an extended period where our hosted metrics pipeline fell behind, causing Grafana dashboards and alerts (including &lt;code&gt;fly-metrics.net&lt;/code&gt;) to show missing or delayed data. The underlying issue was uneven load distribution into our metrics ingestion “aggregator” layer, which led to CPU starvation on a subset of ingestion hosts and large metric queue backlogs that then took significant time to drain. We mitigated by rebalancing ingestion traffic (making the load balancer aware of backend load), tuning queue/throughput settings, and temporarily adding processing capacity to speed up backfill; metrics ingestion resumed and the remaining lag gradually cleared.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Corrosion migration caused elevated CPU and routing failures in BOM and NRT</title>
    <link rel="alternate" href="https://fly.io/infra-log/corrosion-migration-routing/"/>
    <id>https://fly.io/infra-log/corrosion-migration-routing/</id>
    <published>2026-07-02T00:00:00+00:00</published>
    <updated>2026-07-02T12:09:58+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;An ongoing Corrosion database migration on some hosts in BOM and NRT caused SQLite query planner statistics to become stale. As a result, the SQLite query planner chose full table scans for a particular fly-proxy query instead of using the appropriate indexes. This led to high CPU utilization and increased SQLite lock contention within fly-proxy. The resulting long-running read transactions prevented SQLite WAL truncation, further compounding the issue.&lt;/p&gt;

&lt;p&gt;Users experienced HTTP 502 responses and connection resets for traffic routed through the affected regions, and requests to some Machine API endpoints timed out.&lt;/p&gt;

&lt;p&gt;The issue was mitigated by stopping the migration and fly-proxy on the affected hosts, truncating the WAL and running ANALYZE on the affected tables to refresh SQLite&amp;rsquo;s query planner statistics. Once the statistics were refreshed, the query planner resumed using the correct indexes, and fly-proxy was restarted on the affected hosts. The migration script has also been updated to run ANALYZE after migrating each table to prevent this issue from recurring.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Corrosion OOMs and wedged systemd</title>
    <link rel="alternate" href="https://fly.io/infra-log/corrosion-ooms-systemd/"/>
    <id>https://fly.io/infra-log/corrosion-ooms-systemd/</id>
    <published>2026-06-30T00:00:00+00:00</published>
    <updated>2026-06-30T23:01:08+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;Corrosion, our internal state propagation system, was OOM-killed on a small number of hosts. This is usually not a big deal (save for the part where we need to figure out why it exceeded the generous memory limit we gave it), since the &lt;code&gt;systemd&lt;/code&gt; unit is configured to restart the service automatically, and most other services on a host fail gracefully when Corrosion is down temporarily. This time though, the systemd units were stuck &lt;code&gt;activating&lt;/code&gt; with seemingly no progress made, which triggered us to open an internal incident to investigate, and we also quickly increased Corrosion&amp;rsquo;s memory limit on all hosts to prevent any further OOM&amp;rsquo;s.&lt;/p&gt;

&lt;p&gt;This turned out to be related to some &lt;code&gt;fly-proxy&lt;/code&gt; changes we made recently. 2 facts that directly caused this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;code&gt;fly-proxy&lt;/code&gt;&amp;lsquo;s systemd unit contains a &lt;code&gt;After=corrosion.service&lt;/code&gt;;
&lt;/li&gt;&lt;li&gt;&lt;code&gt;fly-proxy&lt;/code&gt; now waits for much longer on shutdown for unfinished connections.
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;When Corrosion got killed this time, it was in the middle of a &lt;code&gt;fly-proxy&lt;/code&gt; rollout, which means that the &lt;code&gt;fly-proxy&lt;/code&gt; unit was still &lt;code&gt;deactivating&lt;/code&gt; on a lot of hosts due to the new shutdown logic. &lt;code&gt;systemd&lt;/code&gt; enforces a &lt;a href='https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html' title=''&gt;reverse order&lt;/a&gt; based on &lt;code&gt;After=&lt;/code&gt; constraints on shutdown, which means that before the last &lt;code&gt;fly-proxy&lt;/code&gt; process stopped, a new &lt;code&gt;corrosion.service&lt;/code&gt; cannot start successfully.&lt;/p&gt;

&lt;p&gt;Since the proxy does not necessarily require a running Corrosion process nowadays thanks to our lazy-loading work, especially not on shutdown, the correct fix here is to simply remove that &lt;code&gt;After=&lt;/code&gt; dependency, which we promptly did. The other loose end in this incident is why Corrosion OOM&amp;rsquo;d in the first place. Our conclusion here is that it is also related to the &lt;code&gt;fly-proxy&lt;/code&gt; rollout: &lt;code&gt;fly-proxy&lt;/code&gt; depends on something called Corrosion &amp;ldquo;updates&amp;rdquo;, which are generated in Corrosion and sent through a Corrosion-internal mpsc channel. The channel became blocked while the proxy restarted, which pushed Corrosion&amp;rsquo;s memory usage over limit, causing the OOM.&lt;/p&gt;

&lt;p&gt;Because Corrosion updates are designed &lt;em&gt;not&lt;/em&gt; to be fully consistent, and &lt;code&gt;fly-proxy&lt;/code&gt; can deal with missed updates gracefully, we shipped &lt;a href='https://github.com/superfly/corrosion/pull/491' title=''&gt;changes&lt;/a&gt; to start dropping updates under this type of pressure. This should prevent this incident from repeating in the future.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Singapore network outage</title>
    <link rel="alternate" href="https://fly.io/infra-log/network-outage-sin/"/>
    <id>https://fly.io/infra-log/network-outage-sin/</id>
    <published>2026-06-25T00:00:00+00:00</published>
    <updated>2026-06-30T17:08:54+00:00</updated>
    <media:thumbnail url="https://fly.io/static/images/default-post-thumbnail.webp"/>
    <content type="html">&lt;p&gt;One of our upstream providers in Singapore region experienced an unexpected failure on a core router. This caused intermittent but complete loss of connectivity for a couple hours on June 18th; the issue reappeared on June 22nd, after which the affected device was replaced.&lt;/p&gt;
</content>
  </entry>
</feed>
