---
title: "Corrosion OOMs and wedged systemd"
---

# Corrosion OOMs and wedged systemd

[← View full Infra Log](https://fly.io/infra-log/)

---

June 23: Corrosion OOMs and wedged systemd (13:45UTC)

Corrosion, our internal state propagation system, was OOM-killed on a small number of hosts. This is usually not a big deal (save for the part where we need to figure out why it exceeded the generous memory limit we gave it), since the `systemd` unit is configured to restart the service automatically, and most other services on a host fail gracefully when Corrosion is down temporarily. This time though, the systemd units were stuck `activating` with seemingly no progress made, which triggered us to open an internal incident to investigate, and we also quickly increased Corrosion’s memory limit on all hosts to prevent any further OOM’s.

This turned out to be related to some `fly-proxy` changes we made recently. 2 facts that directly caused this:

1. `fly-proxy`‘s systemd unit contains a `After=corrosion.service`;
2. `fly-proxy` now waits for much longer on shutdown for unfinished connections.

When Corrosion got killed this time, it was in the middle of a `fly-proxy` rollout, which means that the `fly-proxy` unit was still `deactivating` on a lot of hosts due to the new shutdown logic. `systemd` enforces a [reverse order](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html) based on `After=` constraints on shutdown, which means that before the last `fly-proxy` process stopped, a new `corrosion.service` cannot start successfully.

Since the proxy does not necessarily require a running Corrosion process nowadays thanks to our lazy-loading work, especially not on shutdown, the correct fix here is to simply remove that `After=` dependency, which we promptly did. The other loose end in this incident is why Corrosion OOM’d in the first place. Our conclusion here is that it is also related to the `fly-proxy` rollout: `fly-proxy` depends on something called Corrosion “updates”, which are generated in Corrosion and sent through a Corrosion-internal mpsc channel. The channel became blocked while the proxy restarted, which pushed Corrosion’s memory usage over limit, causing the OOM.

Because Corrosion updates are designed *not* to be fully consistent, and `fly-proxy` can deal with missed updates gracefully, we shipped [changes](https://github.com/superfly/corrosion/pull/491) to start dropping updates under this type of pressure. This should prevent this incident from repeating in the future.