Stateless LLM failover preserves ~0% context; ContinuityBench proxy hits 99.20% CPR
its_vayishu · x · 2026-09-15
The author pushes back on the take that ContinuityBench is "just an LLM API wrapper doing token routing," clarifying they solve different problems:
- Token routing decides which model handles a request for cost, latency, or quality — a real, active research area.
- ContinuityBench targets a different, invisible failure mode: when your primary provider drops and you fail over, a stateless backup never receives prior turns. Nothing crashes, HTTP stays 200, but the model silently forgets the whole conversation while your uptime dashboard stays green.
Across 750 failover events, naive stateless failover preserved close to 0% of context, while their stateful proxy using History Forwarding to reconstruct state at the exact failover point achieved 99.20% CPR (95% CI: 98.27–99.63%). Paper and open benchmark are public.
More from Infra
- Sam Altman: AI query energy cost fell 1,000x-100,000x in 3-4 years — rohanpaul_ai · 2026-11-02
- Marvell and Lumentum to merge, targeting $550B AI connectivity market — BenBajarin · 2026-11-02
- Samsung chairman visits Yokohama advanced packaging lab, pledging ¥40B investment over five years — pstAsiatech · 2026-11-02
- Sam Altman meets Samsung executives in Korea to deepen memory chip and AI infrastructure cooperation — pstAsiatech · 2026-11-02
- Simulating 200,000 months: can your home AI rig pay rent at 2026 GPU prices? — LeviTurk · 2026-11-02
- Indian AI infra firm AM Intelligence seeks up to $3B to buy Nvidia's next-gen chips — pstAsiatech · 2026-11-02