#June 12: Elevated Sprites error rates in SIN (02:27UTC)

Some Sprites API and dashboard requests hitting a SIN edge returned 500s for a subset of orgs with Sprites in the SIN Region. This was caused by a mismatch in code versions running in SIN and SYD regions, as well as the specific way the API handles some organization requests.

In addition to serving general API requests, each API node acts as the ‘manager’ for a subset of organizations in its region. Certain organization request types for some actions will be replayed through the network to that org’s manager node. The ultimate source of truth for this management data lives in object storage. If an API node fails, another node in the region steps in to manage the Sprites the dead node was responsible for. If both nodes in a region fail, API nodes in the next closest region will take up management. So on, so forth.

Earlier in the day one of the two API nodes in SIN failed due to corruption on its local cache volume. The failover process worked as designed, with the other SIN node taking over management of all orgs on the dead node. We brought up a replacement node in SIN and that started re-syncing with the cluster.

While the replacement node was still syncing, the second healthy SIN node restarted. Since it was the only healthy node in the region, this triggered the next closest region (SYD) to take over management of all SIN orgs. Again this succeeded without issue.

However when the two SIN nodes came up, they were unable to replay requests to the nodes in SYD. After some investigation we identified an earlier update had failed to update the SIN nodes to the latest code version. This caused an incompatibility, with the shape of request the SIN nodes were sending not matching what the SYD nodes were expecting. After identifying the issue we pushed out a manual version update to the new SIN nodes and normal operation resumed.

This only impacted requests hitting a SIN region API node, for Sprites that had their management re-homed to the SYD nodes. Requests hitting any other region’s edge for those same Sprites continued working as expected.