AI-generated inference engine hits 1300 tok/s on Xiaomi's 1T model, beating SGLang and vLLM
bingxu_ · x · 2026-09-30
INT21's two-engineer team spent two weeks generating 20 inference engines and benchmarked 15 of them against SGLang and vLLM. Standouts: Xiaomi MiMo V2.6 Pro at 1308 tok/s decode on 8×B300 (vs 540/1011), and DeepSeek V4.1 Flash at 1844 tok/s on 8×B200 — a 2.22×/2.33× speedup. The report, titled "An AlphaGo Moment for Inference?", publishes all measurements and notes limits like higher first-token latency on some workloads.
More from Infra
- Training-free page-sparse decode in llama.cpp: 1.72× faster 128k decoding on an RTX 5090 — topppggg · 2026-11-16
- llama.cpp offloads CI caches and runners to Hugging Face to survive 35 PRs/day — ggerganov · 2026-11-16
- India's Sarvam launches Token Factory, hosting top open models on local infra — HarveenChadha · 2026-11-16
- ZGX Fury stress-tested: 351 tok/s on Qwen3.8-Flash-Next, offline agents and local RAG demos — Sam Witteveen · 2026-11-16
- Gemini 3.5 Flash now runs on air-gapped Google Distributed Cloud, offline up to 12 months — shashib · 2026-11-16
- Gemma.c: full Gemma 1/2/3 multimodal inference in a single C file, ~72 tok/s on CPU at INT8 — This-Purchase-3325 · 2026-11-16