TokenSpeed Refactors KV Cache Architecture
AccBalanced · x · 2026-07-17
TokenSpeed detailed a scheduler refactor: moving away from a memory pool based on a Radix Tree—which became unsuitable as attention patterns grew complex—to a flat, block-based KV cache architecture.
The new implementation uses a single flat paged pool and provides heterogeneous views, making it easier to support different attention mechanisms. The post notes similar approaches in the community, such as vLLM's Jenga and LMDeploy's TurboMind.
This refactor roughly coincided with the release of TML's Inkling, allowing the new architecture to support Inkling from day one. The author also expressed commitment to continuing development with the open-source community.
Related event: Evolution and Refactoring of KV Cache Architecture in LLM Inference(2 posts)→
More from Infra
- Engram's random reads don't suit SSDs; CPU-memory over NVLink could serve all 72 GPUs — bookwormengr · 2026-09-11
- 80% of the DIY LLM inference hype posters have already quit — it's brutally hard systems work — abhijithneil · 2026-09-11
- Hugging Face's Ultra Scale Playbook: a free book on training LLMs on GPU clusters — mdancho84 · 2026-09-11
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11