TokenSpeed: open-source LLM inference engine hits 2.8k stars on GitHub
zhyncs42 · x · 2026-10-26
LightSeek open-sourced TokenSpeed, an LLM inference engine with 2.8k GitHub stars (MIT license). The author explains the scheduler is written in C++ not for performance but correctness: it's built around KV cache state transitions with PD/L2-cache compatibility, needing lossless rollback, which C++ RAII guarantees. The repo includes engine, kernels (AMD/NPU), MLA and scheduler modules, targets Blackwell-class hardware, and supports DeepSeek, GLM, Qwen, Kimi and MiniMax models.
Related event: TokenSpeed Open-Source Inference Engine Explains Its Polyglot Design(2 posts)→
More from coding & agent
- AWS re:Invent 2026 dedicates Security Day to AI: 100+ sessions on securing autonomous agents — DavidLinthicum · 2026-10-26
- Developer uses AI to emulate 1980s hardware synth circuitry — kylegawley · 2026-10-26
- Greptile co-founder on building an AI code review agent: model inversion catches 10% more bugs — AI Engineer · 2026-10-26
- New hire uses company agent to onboard: three prompts from Every — every · 2026-10-26
- Connectome-based memory lets an AI cat recall months of chat in surprising detail — repligate · 2026-10-26
- Theo publishes his Claude skills as tstack; Lockyer jokes about launching dstack — DanielLockyer · 2026-10-26