Stateless LLM failover preserves ~0% context; ContinuityBench proxy hits 99.20% CPR
its_vayishu · x · 2026-09-15
The author pushes back on the take that ContinuityBench is "just an LLM API wrapper doing token routing," clarifying they solve different problems:
- Token routing decides which model handles a request for cost, latency, or quality — a real, active research area.
- ContinuityBench targets a different, invisible failure mode: when your primary provider drops and you fail over, a stateless backup never receives prior turns. Nothing crashes, HTTP stays 200, but the model silently forgets the whole conversation while your uptime dashboard stays green.
Across 750 failover events, naive stateless failover preserved close to 0% of context, while their stateful proxy using History Forwarding to reconstruct state at the exact failover point achieved 99.20% CPR (95% CI: 98.27–99.63%). Paper and open benchmark are public.
More from Infra
- Tuning SGLang on a single 5090 for Qwen3.8-27B: 100 tok/s but only 82k context — ni1by2thetrue · 2026-09-15
- Four dev boards hooked to the internet: test AI-written firmware on real silicon via HTTPS — SelfishlyWandering · 2026-09-15
- Grouped Value Attention shrinks KV cache by reconstructing keys on demand — Vishesh Tripathi · 2026-09-15
- jinfer brings native AI inference to the JVM, matching llama.cpp on CPU with zero Python — mukel90 · 2026-09-15
- Wan 2.2 on one RTX 5090: frame count doesn't touch VRAM, but resolution drops it by 10GB — Realistic-Fennel-190 · 2026-09-15
- 500 trillion tokens in a week: chart points to surging LLM usage — gajesh · 2026-09-15