Together AI's FlashStates hits 3.14x vLLM prefill throughput on coding-agent workloads
burny_tech · x · 2026-11-12
Together AI with UIUC and UT Austin introduces FlashStates, a serving system for hybrid LLMs on agentic workloads. Its Arbitrary State Checkpointing (ASC) decouples recurrent checkpoint positions from KV-cache block boundaries, letting agents resume exactly where they left off, with a memory-constrained cost-aware policy for fine-grained state reuse.
Key results on NVIDIA B200s with coding-agent workloads:
- GLM-5.3-Flash: up to 3.14x vLLM prefill throughput, 88.4% lower p90 TTFT, 3.7x lower latency, 2.5x lower user cost
- Nemotron 3 Super: 2.27x; Qwen3.8-Flash-Next: 1.54x
The authors claim it more than doubles throughput and provider profit for long-conversation agent serving.
More from Infra
- Gratis launches: a local Mac AI assistant with no internet or API fees, built on Bittensor SN65 — markjeffrey · 2026-11-12
- ComfyUI update doubles Qwen LLM token rates on consumer GPUs, up to +251% on RTX 5090 — Lexius2129 · 2026-11-12
- HipKittens turns one: breaking AMD's 'CUDA moat' for a multi-silicon future — simran_s_arora · 2026-11-12
- 'Dear MacKenzie Scott, please send $8M for GPUs' becomes an AI meme — AIFlow_ML · 2026-11-12
- Uno uses discrete diffusion for up to 3x lossless LLM inference speedup, talk Nov 13 — burny_tech · 2026-11-12
- 768 tiny Pollen robots walk a simulated hospital ward on one GPU — MaziyarPanahi · 2026-11-12