FlashStates hits 3.14x vLLM prefill throughput for hybrid-architecture agentic inference

机器之心 · wechat · 2026-11-16

Together AI, UIUC, and UT Austin propose FlashStates, redesigning KV and recurrent-state cache management for hybrid attention+linear-layer models (KDA, GDN, Mamba-2). Integrated into vLLM; SGLang has adopted its cost-aware eviction policy.

Problem: Agentic multi-turn loads require both full KV and a recurrent checkpoint exactly at the resume point. Analysis of real Claude Code traces finds three avoidable miss classes: placement misses (43% of uncached tokens on vLLM at low concurrency), dependency misses (SGLang's LRU evicts checkpoints whose KV survives), and capacity misses — checkpoint density alone can't fix this.

Design:

Results (B200, real traces): prefill throughput up to 3.15x vLLM (GLM-5.3-Flash), 2.27x (Nemotron 3 Super), 1.54x (Qwen3.8-Flash-Next); p90 TTFT down up to 88.4%. With a 512GiB Mooncake CPU tier, output throughput gains another 41.4% over vLLM+Mooncake, and p90 TTFT at 80 sessions drops from 22s to 7.18s, with cross-engine/node state reuse enabling PD-disaggregated deployments.

Original post →

More from Infra

Infra channel →