Metastable Failures Revisited: Why the Microsoft Outage Kept Relapsing Under Traffic
tianyin_xu · x · 2026-11-15
Researcher Aleksey Charapko revisits metastable failure recovery in light of a recent Microsoft outage where restored servers became unresponsive again once traffic returned — a classic signature of metastability.
How metastable failures work:
- An initial trigger (CPU/I/O/lock overload) throws the system off balance
- An amplification loop sustains the failure: requests time out under overload, clients retry, retries add more load, causing more timeouts — persisting even after the original fault is fixed
Recovery requires two things:
- Breaking the amplification loop (stop digging deeper)
- Dealing with the accumulated backlog of pending work (often by dropping load)
Rebooting 'works by luck': it clears the backlog that fuels the feedback loop, though cold-starting large systems too quickly can itself trigger a new metastable failure. The author stresses his analysis is informed speculation based on the public incident report.
More from Infra
- ARK analyst: future AI bottleneck won't be power but reusable launch capacity — DMaguireARK · 2026-11-15
- Wyoming could fund its entire $1.7B state budget from data center revenues — panickssery · 2026-11-15
- FreeToken open-source engine runs 284B DeepSeek locally at 25 tok/s on RTX 5090 — aigclink · 2026-11-15
- WSJ: Southeast Asia data centers supplied Chinese firms compute and covered chip smuggling — BenBajarin · 2026-11-15
- Parallel Prompting speeds shared-context LLM inference up to 6x in SGLang — kalyan_kpl · 2026-11-15
- JP Morgan private bank publishes note on Nvidia's Jensen Huang — BenBajarin · 2026-11-15