#June 23: Corrosion OOMs and wedged systemd (13:45UTC)

Corrosion, our internal state propagation system, was OOM-killed on a small number of hosts. This is usually not a big deal (save for the part where we need to figure out why it exceeded the generous memory limit we gave it), since the systemd unit is configured to restart the service automatically, and most other services on a host fail gracefully when Corrosion is down temporarily. This time though, the systemd units were stuck activating with seemingly no progress made, which triggered us to open an internal incident to investigate, and we also quickly increased Corrosion’s memory limit on all hosts to prevent any further OOM’s.

This turned out to be related to some fly-proxy changes we made recently. 2 facts that directly caused this:

1. fly-proxy‘s systemd unit contains a After=corrosion.service;
2. fly-proxy now waits for much longer on shutdown for unfinished connections.

When Corrosion got killed this time, it was in the middle of a fly-proxy rollout, which means that the fly-proxy unit was still deactivating on a lot of hosts due to the new shutdown logic. systemd enforces a reverse order based on After= constraints on shutdown, which means that before the last fly-proxy process stopped, a new corrosion.service cannot start successfully.

Since the proxy does not necessarily require a running Corrosion process nowadays thanks to our lazy-loading work, especially not on shutdown, the correct fix here is to simply remove that After= dependency, which we promptly did. The other loose end in this incident is why Corrosion OOM’d in the first place. Our conclusion here is that it is also related to the fly-proxy rollout: fly-proxy depends on something called Corrosion “updates”, which are generated in Corrosion and sent through a Corrosion-internal mpsc channel. The channel became blocked while the proxy restarted, which pushed Corrosion’s memory usage over limit, causing the OOM.

Because Corrosion updates are designed not to be fully consistent, and fly-proxy can deal with missed updates gracefully, we shipped changes to start dropping updates under this type of pressure. This should prevent this incident from repeating in the future.