#August 8: Global Corrosion lag caused Machines API app-not-found errors (23:52UTC)

A subset of very large service-definition updates from machines with large numbers of services could not be applied to our global service-discovery database (Corrosion) within the configured timeout on some nodes. These updates were repeatedly retried, slowing down or blocking the application of other updates. Since the Machines API (via Flaps) reads app metadata from global Corrosion, some newly created apps and recent updates were not visible everywhere, resulting in elevated app-not-found errors.

We mitigated the issue by adjusting the timeout and Corrosion’s retry behavior so that failed updates do not retry indefinitely and block other updates. We also rolled out a fix to chunk updates when a machine has a large number of services, preventing such updates from causing similar propagation issues in the future.