Microsoft 365 outage postmortem: how automated health checks amplified the failure
_jaydeepkarale · x · 2026-11-16
A postmortem-style thread on the recent Microsoft 365 outage explains how the very mechanism designed to protect production — automated health checks — ended up amplifying the incident.
- In distributed systems, servers failing readiness checks get removed from the traffic pool, which is exactly what should happen; but during this outage, the component processing those health checks itself started failing.
- Affected servers kept failing automated checks and were pulled out; Microsoft tested an alternative mechanism to bring some back online.
- The cascade: fewer available servers → more traffic on the rest → higher load → more failures.
The takeaway: even protective automation needs failure-mode design, or it can worsen the very incident it's meant to prevent.
More from Infra
- Commerce Dept set to release remote access rule tonight, reporter says — ShakeelHashim · 2026-11-16
- Liquid AI d1-2B decision model runs on-device: ~68ms gesture reading, 14 decisions/s, zero tokens — AccBalanced · 2026-11-16
- Microsoft Outlook outage silenced chlorine alert emails at German theme park, injuring 15 pool guests — AccBalanced · 2026-11-16
- Ming-Chi Kuo: OpenAI and Anthropic orders extend Google TPU visibility to 2029, ~26M units — BenBajarin · 2026-11-16
- Cerebras core revenue up 70% YoY to $231M, backlog hits $26.1B — AccBalanced · 2026-11-16
- Nvidia CEO saga: firstadopter discusses Jensen Huang's resignation and successor race on TBPN — firstadopter · 2026-11-16