#June 25: Corrosion migration caused elevated CPU and routing failures in BOM and NRT (12:06UTC)

An ongoing Corrosion database migration on some hosts in BOM and NRT caused SQLite query planner statistics to become stale. As a result, the SQLite query planner chose full table scans for a particular fly-proxy query instead of using the appropriate indexes. This led to high CPU utilization and increased SQLite lock contention within fly-proxy. The resulting long-running read transactions prevented SQLite WAL truncation, further compounding the issue.

Users experienced HTTP 502 responses and connection resets for traffic routed through the affected regions, and requests to some Machine API endpoints timed out.

The issue was mitigated by stopping the migration and fly-proxy on the affected hosts, truncating the WAL and running ANALYZE on the affected tables to refresh SQLite’s query planner statistics. Once the statistics were refreshed, the query planner resumed using the correct indexes, and fly-proxy was restarted on the affected hosts. The migration script has also been updated to run ANALYZE after migrating each table to prevent this issue from recurring.