Aligning KV-cache blocks with agent turns yields up to 3x throughput gains
heghbalz · x · 2026-11-10
The poster is stunned that simply making KV-cache block boundaries aware of agent turns—so mid-block tokens aren't evicted—delivers massive efficiency gains: "we are soo early."
The attached benchmark chart compares FlashState vs vLLM across three models at 24-48 concurrent sessions:
- GLM-5.3-Flash NVFP4 (1M ctx): decode TPM/GPU up 3.02x, prefill up 3.14x, uncached tokens p50 down 96%, TTFT p90 down 88%
- Nemotron 3 Super (1M): decode up 2.32x, uncached tokens down 97%, TTFT down 77%
- Qwen3.8-Flash-Next (512K): both up 1.54x, uncached tokens down 66%, TTFT down 79%
Key insight: aligning cache chunking with the multi-turn structure of agent workloads dramatically improves cache hit rates—an overlooked optimization in today's inference stacks.
More from coding & agent
- A first-principles beginner's guide to context engineering for agents — sharpeye_wnl · 2026-11-10
- Orchestrating multiple agents with a shared agents.md and topic workspaces — _AustinCalvert_ · 2026-11-10
- Bypass Grok Bot Usage Limits: Hook Up T3 Code and Have It Call Opus 5.5 — Rasmic · 2026-11-10
- Hark uses animated stickers to show agent task status on iMessage and RCS — _AustinCalvert_ · 2026-11-10
- He Offloaded Claude Code's File Reading to Gemini via an MCP Server, Cutting Context 99% — Lucanhtai15 · 2026-11-10
- 'Tell your agent to find all implicit state machines' — and neither dev knows what one is — RajaswaPatil · 2026-11-10