TokenSpeed Refactors KV Cache Architecture
AccBalanced · x · 2026-07-17
TokenSpeed detailed a scheduler refactor: moving away from a memory pool based on a **Radix Tree**—which became unsuitable as attention patterns grew complex—to a **flat, block-based KV cache architecture**. The new implementation uses a single flat paged pool and provides heterogeneous views, making it easier to support different attention mechanisms. The post notes similar approaches in the community, such as **vLLM's Jenga** and **LMDeploy's TurboMind**. This refactor roughly coincided with the release of TML's **Inkling**, allowing the new architecture to support Inkling from day one. The author also expressed commitment to continuing development with the open-source community.
Related event: Evolution and Refactoring of KV Cache Architecture in LLM Inference(2 posts)→
More from Infra
- Local AI may pay back in 6–7 years and cut long-term costs by 30–40% — DavidLinthicum · 2026-07-21
- TSMC reportedly plans up to 10% chipmaking price hikes in 2027 — kimmonismus · 2026-07-21
- More open models and llama.cpp updates are coming, says Merve Noyan — mervenoyann · 2026-07-21
- Why adding a second LLM provider breaks more than the API surface — Ok_Extension6373 · 2026-07-21
- UK AI datacentres face backlash over heat, noise and land use — nordicinst · 2026-07-21
- Fluidstack raises $830M at $7.5B valuation as Anthropic backs a $50B compute buildout — rohanpaul_ai · 2026-07-21