LightSpec: Open-Sourcing Universal Dynamic MTP Framework
机器之心 · wechat · 2026-07-16
The LightLLM team has open-sourced LightSpec, a universal dynamic MTP inference framework, aiming to evolve dynamic multi-token prediction from a model-specific trick into a universal inference infrastructure.\n\nIts core is a two-stage, training-free runtime planner: it first dynamically determines the verification budget for the current round, and then sets the draft budget for the next. Instead of relying on a separately trained confidence model, scheduling leverages naturally generated timing tables and EMA acceptance rates during inference for online decisions. This ensures compatibility with different draft models like DSpark and Eagle3.\n\nThe article also details the full system implementation, featuring runtime cost tables, online acceptance stats, dynamic input construction, and pipeline coordination to maximize CPU scheduling and GPU execution overlap. Experiments show that under Qwen3-14B, H200, and 256 concurrency, this dynamic approach boosts throughput across various workloads, with particularly notable advantages at larger MTP step sizes.
More from Infra
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11
- Can a 7900 XTX 24GB run Qwen locally? Reddit seeks ROCm tok/s benchmarks — thenomadexplorerlife · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11