August 24: Metrics backlog

#August 24: Metrics backlog (09:59UTC)

A surge of hot & fresh time-series blew out our storage/query cardinality and caused metrics to appear delayed by 30-60 minutes. We spun up some short term metrics capacity to absorb the load and caught up on the backlog without dropping anything along the way. Once caught up, the metrics cluster has been ticking away happily, albeit at its presumably anomalous new volume. (We’ll get to that soon.)