TokenSpeed hits 331 tok/s per GPU serving Kimi K3 on 32 GB300s, 1.92× baseline
zhyncs42 · x · 2026-11-11
LightSeek's TokenSpeed serves Kimi K3 NVFP4 at 331.18 output tokens/s per GPU on the full SemiAnalysis AgentX trace using 32 NVIDIA GB300 GPUs—1.92× the per-GPU throughput of a standard eight-GPU DCP8 baseline.
Key design: independent DWDP16 prefill + efficient DEP16 batched decode, a shared cache retaining agent histories across turns (freeing prefill placement from conversation affinity), and async scheduling so prep/transfer never interrupts useful GPU work. The core thesis: cache hit rate and workload balance are the foundations of high-throughput agentic inference. Kimi K3 is a 2.8T-param MoE (104B active) whose size makes HBM cache capacity the main serving constraint.
More from Infra
- AMD Zen 6 deep dive: cheaper branch mispredicts and a return to Zen 3/4 integer scheduling — jedisct1 · 2026-11-11
- Together AI SVP Dan Fu on kernels, serving 3T-param models day zero, and self-improving inference — togethercompute · 2026-11-11
- ExecuTorch adds WebGPU backend: run PyTorch models in-browser, up to 10x faster than CPU — PyTorch · 2026-11-11
- Hedra runs training experiments with AI agents steered by human researchers — testingcatalog · 2026-11-11
- Simular Box gives Claude, ChatGPT and Cursor agents their own cloud Linux desktop via MCP — AdFluid9823 · 2026-11-11
- Polymarket: 25% chance a US state enacts a data center moratorium this year — Polymarket · 2026-11-11