FlashStates hits 3.14x vLLM prefill throughput for hybrid-architecture agentic inference
机器之心 · wechat · 2026-11-16
Together AI, UIUC, and UT Austin propose FlashStates, redesigning KV and recurrent-state cache management for hybrid attention+linear-layer models (KDA, GDN, Mamba-2). Integrated into vLLM; SGLang has adopted its cost-aware eviction policy.
Problem: Agentic multi-turn loads require both full KV and a recurrent checkpoint exactly at the resume point. Analysis of real Claude Code traces finds three avoidable miss classes: placement misses (43% of uncached tokens on vLLM at low concurrency), dependency misses (SGLang's LRU evicts checkpoints whose KV survives), and capacity misses — checkpoint density alone can't fix this.
Design:
- Arbitrary-position state checkpoints (ASC), captured at exact request endpoints, with a PrefillExport interface for mid-prefill export
- A dependency-aware Greedy-Dual-Size admission/eviction policy minimizing total re-prefill time, evicting checkpoint + exclusively-depended blocks with no load-specific hyperparameters
Results (B200, real traces): prefill throughput up to 3.15x vLLM (GLM-5.3-Flash), 2.27x (Nemotron 3 Super), 1.54x (Qwen3.8-Flash-Next); p90 TTFT down up to 88.4%. With a 512GiB Mooncake CPU tier, output throughput gains another 41.4% over vLLM+Mooncake, and p90 TTFT at 80 sessions drops from 22s to 7.18s, with cross-engine/node state reuse enabling PD-disaggregated deployments.
More from Infra
- Nvidia scandal isn't slowing down: Huang's near-$100B power web under strain — The Verge AI · 2026-11-16
- Volume rules chips: halvarflake says Intel lost by outsourcing demand — TheKanter · 2026-11-16
- Taipei 101 lights up "Taiwan ❤️ Jensen" as tech sector rallies behind Jensen Huang — liu_mingyu · 2026-11-16
- Nous exec: avoid GPT 900k context variant — extra charges kick in past 272k — Teknium · 2026-11-16
- CUDA matmul optimization walkthrough: naive kernel to 93.7% of cuBLAS — RisingSayak · 2026-11-16
- Dev runs 120B models on iOS by streaming MoE weights from flash in llama.cpp — Chance-Resource-4970 · 2026-11-16