Trailing 2 Weeks Incidents
(Larger boxes are longer, darker boxes are higher-impact; see last week’s bulletin for details about the top row of incidents).
- November 12: Network Disruption in Querétaro (14:00EST): We experienced a complete network outage (couldn’t reach anything in our DC), which cleared up inside of 3 minutes, followed by almost 4 hours of sporadic severe routing disruption (latency, packet drops, occasional loss of connectivity), sufficient to disrupt orchestration in this region. The problem was traced to an upstream of our upstream in Mexico.
This Week In Engineering
Despite a quiet week incident-wise, the whole team was unusually interrupt-driven this week; a consequence of catching a bunch of stuff before it could actually become an incident.