TokenSpeed Refactors KV Cache Architecture
AccBalanced · x · 2026-07-17
TokenSpeed detailed a scheduler refactor: moving away from a memory pool based on a Radix Tree—which became unsuitable as attention patterns grew complex—to a flat, block-based KV cache architecture.
The new implementation uses a single flat paged pool and provides heterogeneous views, making it easier to support different attention mechanisms. The post notes similar approaches in the community, such as vLLM's Jenga and LMDeploy's TurboMind.
This refactor roughly coincided with the release of TML's Inkling, allowing the new architecture to support Inkling from day one. The author also expressed commitment to continuing development with the open-source community.
Related event: Evolution and Refactoring of KV Cache Architecture in LLM Inference(2 posts)→
More from Infra
- Revenue per GPU can rise as token prices fall, argues Gavin Baker — GavinSBaker · 2026-10-27
- Anthropic S-1: only $19B total compute spend since inception, overbuild warning — bookwormengr · 2026-10-27
- CoreWeave, You.com and NVIDIA lift BrowseComp 22.9% in 3-week RL post-training sprint — RichardSocher · 2026-10-27
- Grid queues make power architecture an AI-factory decision; Exowatt and Redwood offer fixes — ingliguori · 2026-10-27
- Two CVSS 9.9 Wasmtime flaws allow sandbox escape, one ARM64-only since 2020 — basedjensen · 2026-10-27
- AI Infra Meme: While Countries Need TW-Scale Power for Datacenters, People Argue About Lightbulbs — mgostIH · 2026-10-27