Probes stayed green while users errored: how to detect LLM provider outages earlier
Suresh-R · reddit · 2026-11-04
An LLM gateway developer shares how synthetic probes missed two real provider incidents and asks the community for better alerting strategies.
- Root cause: the probe sent one small request on a fixed path, while failures hit real traffic with a different request shape — canary said healthy while users were already erroring.
- Current setup: layered monitoring with per-provider error-rate and latency-percentile alerts on real traffic, keeping the probe only as a last-resort signal.
- Open problem: threshold tuning — alerting on the first failed requests pages on noise; waiting for a rate window confirms the outage minutes late.
- Asks how others draw the line between probe-based, traffic-based, and status-page signals, and how many minutes of real failures they tolerate before paging.
Directly useful for teams operating LLM gateways or multi-provider routing.
More from Infra
- Palmer Luckey's winning data center formula: no subsidies, own power, waterless, silent — NinaDSchick · 2026-11-04
- xgboost-metal open-sources Apple GPU-accelerated XGBoost via Metal port — HankYeomans · 2026-11-04
- Ex-Amazon insider: AWS earns higher margins on open-weight tokens than Claude — RihardJarc · 2026-11-04
- PE fund to pour $2B into a neocloud; SemiAnalysis vets its frontier-lab contract — AccBalanced · 2026-11-04
- DeepSeek serves only 17% of its own tokens as 19+ inference providers split the rest — AccBalanced · 2026-11-04
- A month at NetBird: agents devs ignore the networking layer at their peril — victorialslocum · 2026-11-04