#July 3: Power supply failure in ORD (00:07UTC)

For redundancy, servers generally have two power supplies, connected to two independent power feeds: if one power supply fails, or one power feed goes down, the server can keep operating on the other power supply. During normal operation, the load is shared between both supplies.

Our servers are, of course, no exception. However, when one power feed of an ORD datacenter went down, we found our servers’ CPUs heavily throttled, to the point where they were unable to do any useful work. Later discussing with the provider, we found a misconfigured setting: the server would throttle the CPUs when one power supply failed or lost power.

Now, this isn’t a “oh duh, it should obviously not be like that” situation: this is a safety feature designed to not overload one power feed. Single-power feed operation, in some cases, can just be a fallback for the server to shut down safely without the ability to support normal tasks, even though this is not true in our case – we expect it to provide better uptime at full performance. The throttling setting makes sense as a safe default, and we should be explicitly opting out of the safety feature when we are sure single-power feed operation is safe in our case.

As a result of this incident, we are now working with our providers to ensure all of our servers’ settings are configured best for how our power feeds are set up.