--- title: "Infra Log" description: "The home for Infra Engineering content on Fly.io." --- # Infra Log 135 pages, newest first. Each link is the page's Markdown twin; the page itself is the same URL without `.md`, ending in `/`. - [Sprites API partial outage from SQLite database evictions](https://fly.io/infra-log/sprites-api-sqlite-evictions.md) - [Sidekiq periodic scheduler not running after Redis timeout](https://fly.io/infra-log/sidekiq-scheduler-redis.md) - [Fly-proxy HTTP/2 routing broke due to missing Consul service rows](https://fly.io/infra-log/http2-routing-consul.md) - [Stuck background job queue blocked Sprite deletions](https://fly.io/infra-log/job-queue-blocked.md) - [WireGuard gateways broke after schema migration mismatch](https://fly.io/infra-log/wireguard-schema-mismatch.md) - [Metrics backlog](https://fly.io/infra-log/metrics-backlog.md) - [Development node advertised Fly public DNS anycast IPs and broke it](https://fly.io/infra-log/dns-anycast-break.md) - [MPGv1 clusters in ORD-0 failed to restart](https://fly.io/infra-log/mpgv1-restart-ord.md) - [Some API endpoints briefly rejected flyctl requests](https://fly.io/infra-log/api-rejected-flyctl.md) - [Petsem primary crash loop from corrupted volume](https://fly.io/infra-log/petsem-crashloop-volume.md) - [Stuck nftables reload blocked flyd startup causing API errors](https://fly.io/infra-log/nftables-reload-flyd.md) - [Egress IP issues in SJC](https://fly.io/infra-log/egress-ip-sjc.md) - [Global Corrosion lag caused Machines API app-not-found errors](https://fly.io/infra-log/machines-api-errors.md) - [BGP misconfiguration dropped ingress traffic](https://fly.io/infra-log/bgp-ingress-drop.md) - [Certificate issuance delays from Sidekiq overload](https://fly.io/infra-log/sidekiq-cert-delays.md) - [IAD MPGv2 Patroni etcd lease-expiration storm](https://fly.io/infra-log/patroni-etcd-storm.md) - [Missing app logs because we forgot a `systemctl restart`](https://fly.io/infra-log/missing-app-logs.md) - [tkdb primary host outage caused token validation failures](https://fly.io/infra-log/tkdb-primary-outage.md) - [Vault throws one last tantrum on its way out the door](https://fly.io/infra-log/vault-tantrum.md) - [SJC worker hosts locked up and rebooted](https://fly.io/infra-log/worker-lockups-sjc.md) - [A couple of deployment issues](https://fly.io/infra-log/deployment-issues.md) - [Certificate issuance outage](https://fly.io/infra-log/certificate-issuance-outage.md) - [Power supply failure in ORD](https://fly.io/infra-log/power-failure-ord.md) - [Runaway cleanup job filled Redis](https://fly.io/infra-log/redis-cleanup-job.md) - [VictoriaMetrics ingestion delays and backlog](https://fly.io/infra-log/victoriametrics-ingestion-delays.md) - [Corrosion migration caused elevated CPU and routing failures in BOM and NRT](https://fly.io/infra-log/corrosion-migration-routing.md) - [Corrosion OOMs and wedged systemd](https://fly.io/infra-log/corrosion-ooms-systemd.md) - [Machines API Outage](https://fly.io/infra-log/machines-api-outage.md) - [Singapore network outage](https://fly.io/infra-log/network-outage-sin.md) - [Elevated Sprites error rates in SIN](https://fly.io/infra-log/sprites-errors-sin.md) - [Everyone lives on NULL island](https://fly.io/infra-log/null-island.md) - [ORD transit loss destabilized Managed Postgres clusters](https://fly.io/infra-log/transit-loss-ord.md) - [Two GRU edges OOM](https://fly.io/infra-log/gru-edges-oom.md) - [Stale 6PN mappings wreaking havoc](https://fly.io/infra-log/stale-6pn-mappings.md) - [Deploys blocked by billing error](https://fly.io/infra-log/deploys-billing-error.md) - [DNS cache was broken for CNAME'd domains](https://fly.io/infra-log/dns-cache-cname.md) - [West coast edge proxies overloaded](https://fly.io/infra-log/edge-proxies-overload.md) - [App creation timeouts from petsem-certs disk full](https://fly.io/infra-log/petsem-certs-diskfull.md) - [Fly dashboard outage from broken GraphQL API deploy](https://fly.io/infra-log/graphql-deploy-outage.md) - [SYD egress IP networking broken on new workers](https://fly.io/infra-log/egress-ip-syd.md) - [IAD observability disrupted by NATS misconfiguration](https://fly.io/infra-log/nats-misconfig-iad.md) - [Proxy and Corrosion in SIN weren’t on the same page](https://fly.io/infra-log/proxy-corrosion-sin.md) - [FRA managed Postgres control-plane outage](https://fly.io/infra-log/postgres-control-plane-outage.md) - [Bad TLS cert update broke Consul](https://fly.io/infra-log/tls-cert-consul.md) - [Usage ingestion blocked by stuck Oban jobs](https://fly.io/infra-log/oban-jobs-stuck.md) - [Kernel upgrade caused machine `stdout` to become wedged by Cloud Hypervisor](https://fly.io/infra-log/cloud-hypervisor-stdout.md) - [Machines API hitting failed hosts in SIN](https://fly.io/infra-log/machines-api-sin.md) - [Petsem primary host lost networking (IAD)](https://fly.io/infra-log/primary-host-networking-iad.md) - [Machines API bug caused `fly deploy` to create duplicate Machines](https://fly.io/infra-log/machines-api-duplicates.md) - [GitHub integration management callbacks returned 500s on Fly.io secondary nodes](https://fly.io/infra-log/github-callbacks-500s.md) - [MPG provisioning failures from revoked org token](https://fly.io/infra-log/mpg-provisioning-token.md) - [Extension provider polling overloaded Postgres](https://fly.io/infra-log/provider-polling-postgres.md) - [Duplicate Wireguard Mesh IPs Wreaking Havoc](https://fly.io/infra-log/wireguard-ip-duplicates.md) - [Vault outage broke TLS certificate lookups](https://fly.io/infra-log/vault-tls-outage.md) - [WireGuard wg0 one-way host connectivity](https://fly.io/infra-log/wireguard-one-way.md) - [High edge CPU usage resulting in high latency in ORD](https://fly.io/infra-log/edge-cpu-ord.md) - [NRT Machines API errors thrashed Managed Postgres](https://fly.io/infra-log/machines-api-nrt.md) - [SYD host I/O saturation](https://fly.io/infra-log/host-io-saturation.md) - [Private networking outage in Sydney (again)](https://fly.io/infra-log/private-networking-syd.md) - [Web Sidekiq backlog from stuck usage jobs](https://fly.io/infra-log/sidekiq-backlog-usage.md) - [Sprite creation errors in SJC and AMS](https://fly.io/infra-log/sprite-errors-sjc-ams.md) - [IAD CPU crunch](https://fly.io/infra-log/cpu-crunch-iad.md) - [Sprites API errors](https://fly.io/infra-log/sprites-api-errors.md) - [FRA region outage](https://fly.io/infra-log/fra-outage.md) - [ORD machine creates bogged down](https://fly.io/infra-log/machine-creates-ord.md) - [GraphQL timeouts](https://fly.io/infra-log/graphql-timeouts.md) - [Errors viewing logs in Grafana](https://fly.io/infra-log/grafana-log-errors.md) - [DFW capacity, again](https://fly.io/infra-log/dfw-capacity.md) - [Metrics outage](https://fly.io/infra-log/metrics-outage.md) - [DFW capacity errors](https://fly.io/infra-log/capacity-errors-dfw.md) - [SJC disappeared (briefly)](https://fly.io/infra-log/sjc-disappearance.md) - [A bunch of wedged Sprites](https://fly.io/infra-log/wedged-sprites.md) - [Tight capacity in ORD and SIN](https://fly.io/infra-log/capacity-ord-sin.md) - [Sprites API didn't like numbers](https://fly.io/infra-log/sprites-api-numbers.md) - [GraphQL overloaded](https://fly.io/infra-log/graphql-overload.md) - [Sydney WireGuard outages from upstream UDP filtering](https://fly.io/infra-log/wireguard-udp-filtering.md) - [App-scoped IPs deleted by mistake](https://fly.io/infra-log/app-ips-deleted.md) - [BGP route leak sent North America traffic to Singapore](https://fly.io/infra-log/bgp-route-leak.md) - [GraphQL mutations failing](https://fly.io/infra-log/graphql-mutations.md) - [Petsem got overwhelmed](https://fly.io/infra-log/petsem-overload.md) - [Cost Explorer errors from internal timeouts](https://fly.io/infra-log/cost-explorer-timeouts.md) - [Our certs vault went down and took some proxies with it](https://fly.io/infra-log/our-certs-vault-went-down.md) - [Infra Log: Season 2](https://fly.io/infra-log/season-two.md) - [2025-05-31](https://fly.io/infra-log/2025-05-31.md) - [2025-05-24](https://fly.io/infra-log/2025-05-24.md) - [2025-05-17](https://fly.io/infra-log/2025-05-17.md) - [2025-05-10](https://fly.io/infra-log/2025-05-10.md) - [2025-05-03](https://fly.io/infra-log/2025-05-03.md) - [2025-04-26](https://fly.io/infra-log/2025-04-26.md) - [2025-04-19](https://fly.io/infra-log/2025-04-19.md) - [2025-04-12](https://fly.io/infra-log/2025-04-12.md) - [2025-04-05](https://fly.io/infra-log/2025-04-05.md) - [2025-03-22](https://fly.io/infra-log/2025-03-22.md) - [2025-03-15](https://fly.io/infra-log/2025-03-15.md) - [2025-03-01](https://fly.io/infra-log/2025-03-01.md) - [2025-02-22](https://fly.io/infra-log/2025-02-22.md) - [2025-02-15](https://fly.io/infra-log/2025-02-15.md) - [2025-02-08](https://fly.io/infra-log/2025-02-08.md) - [2025-02-01](https://fly.io/infra-log/2025-02-01.md) - [2025-01-25](https://fly.io/infra-log/2025-01-25.md) - [2025-01-18](https://fly.io/infra-log/2025-01-18.md) - [2025-01-11](https://fly.io/infra-log/2025-01-11.md) - [2025-01-04](https://fly.io/infra-log/2025-01-04.md) - [2024-12-14](https://fly.io/infra-log/2024-12-14.md) - [2024-12-07](https://fly.io/infra-log/2024-12-07.md) - [2024-11-30](https://fly.io/infra-log/2024-11-30.md) - [2024-11-16](https://fly.io/infra-log/2024-11-16.md) - [2024-11-09](https://fly.io/infra-log/2024-11-09.md) - [2024-11-02](https://fly.io/infra-log/2024-11-02.md) - [2024-10-26](https://fly.io/infra-log/2024-10-26.md) - [2024-10-19](https://fly.io/infra-log/2024-10-19.md) - [2024-10-12](https://fly.io/infra-log/2024-10-12.md) - [2024-10-05](https://fly.io/infra-log/2024-10-05.md) - [2024-09-28](https://fly.io/infra-log/2024-09-28.md) - [2024-09-21](https://fly.io/infra-log/2024-09-21.md) - [2024-09-14](https://fly.io/infra-log/2024-09-14.md) - [2024-09-07](https://fly.io/infra-log/2024-09-07.md) - [2024-08-31](https://fly.io/infra-log/2024-08-31.md) - [2024-08-24](https://fly.io/infra-log/2024-08-24.md) - [2024-08-17](https://fly.io/infra-log/2024-08-17.md) - [2024-08-10](https://fly.io/infra-log/2024-08-10.md) - [2024-08-03](https://fly.io/infra-log/2024-08-03.md) - [2024-07-27](https://fly.io/infra-log/2024-07-27.md) - [2024-07-20](https://fly.io/infra-log/2024-07-20.md) - [2024-07-13](https://fly.io/infra-log/2024-07-13.md) - [2024-07-06](https://fly.io/infra-log/2024-07-06.md) - [2024-06-29](https://fly.io/infra-log/2024-06-29.md) - [2024-06-22](https://fly.io/infra-log/2024-06-22.md) - [2024-06-15](https://fly.io/infra-log/2024-06-15.md) - [2024-06-08](https://fly.io/infra-log/2024-06-08.md) - [2024-06-01](https://fly.io/infra-log/2024-06-01.md) - [2024-05-25](https://fly.io/infra-log/2024-05-25.md) - [2024-05-18](https://fly.io/infra-log/2024-05-18.md) - [2024-05-11](https://fly.io/infra-log/2024-05-11.md) - [2024-05-05](https://fly.io/infra-log/2024-05-05.md)