ContinuityBench: Stateful Failover Preserves LLM Conversations at 99.2% Across Providers
its_vayishu · x · 2026-10-15
A new arXiv paper, ContinuityBench (Pandey & Singh), tackles stateful failover in multi-provider LLM routing: high API availability does not equal conversational continuity—stateless failover keeps uptime but silently discards conversation history.
- Introduces two metrics: Continuity Preservation Rate (CPR) and Continuity Latency Overhead (CLO).
- Proposes a stateful multi-provider proxy using History-Forwarding to reconstruct conversational state across heterogeneous LLM endpoints.
- Releases continuity-bench, an open harness to stress-test context preservation under high-concurrency failures.
- Across 750 failover events, the stateful proxy achieved 99.20% CPR (95% CI 98.27–99.63%) vs near-0% for stateless architectures.
- Also characterizes failover latency, highlighting the need for asynchronous exponential backoff.
Metriqual productizes this architecture for routing, failover, and observability.
More from Infra
- Gumroad cut $100k/year from AWS by moving one Rails webhook endpoint to Go on 3 servers — DanielLockyer · 2026-10-15
- Anthropic's margins imply 70-75% of its compute goes to R&D, analysis shows — Jsevillamol · 2026-10-15
- First PyTorch Madrid meetup demos llm-d: K8s-native distributed LLM inference on vLLM — SergioPaniego · 2026-10-15
- Tsinghua's Latent Compaction Compresses KV Cache 32× While Keeping 96.7% Retrieval Accuracy — Yingfa Chen · 2026-10-15
- Starlink Wi-Fi spotted on Austrian Airlines, speedtest shows 157 Mbps down at 43ms ping — mitsuhiko · 2026-10-15
- Banks try to offload AI data center debt they just underwrote — jazir55 · 2026-10-15