#June 28: VictoriaMetrics ingestion delays and backlog (08:00UTC)

We had an extended period where our hosted metrics pipeline fell behind, causing Grafana dashboards and alerts (including fly-metrics.net) to show missing or delayed data. The underlying issue was uneven load distribution into our metrics ingestion “aggregator” layer, which led to CPU starvation on a subset of ingestion hosts and large metric queue backlogs that then took significant time to drain. We mitigated by rebalancing ingestion traffic (making the load balancer aware of backend load), tuning queue/throughput settings, and temporarily adding processing capacity to speed up backfill; metrics ingestion resumed and the remaining lag gradually cleared.