LightSpec: Open-Sourcing Universal Dynamic MTP Framework
机器之心 · wechat · 2026-07-16
The LightLLM team has open-sourced LightSpec, a universal dynamic MTP inference framework, aiming to evolve dynamic multi-token prediction from a model-specific trick into a universal inference infrastructure.\n\nIts core is a two-stage, training-free runtime planner: it first dynamically determines the verification budget for the current round, and then sets the draft budget for the next. Instead of relying on a separately trained confidence model, scheduling leverages naturally generated timing tables and EMA acceptance rates during inference for online decisions. This ensures compatibility with different draft models like DSpark and Eagle3.\n\nThe article also details the full system implementation, featuring runtime cost tables, online acceptance stats, dynamic input construction, and pipeline coordination to maximize CPU scheduling and GPU execution overlap. Experiments show that under Qwen3-14B, H200, and 256 concurrency, this dynamic approach boosts throughput across various workloads, with particularly notable advantages at larger MTP step sizes.
More from Infra
- oMLX 0.5.2 adds Mac menu-bar stats, low-bit decode kernels, and faster downloads — awnihannun · 2026-07-22
- Strangeworks launches Aura to turn enterprise ops into production optimization systems — whurley · 2026-07-22
- Graph workload 854.graph500 enters SPEC CPU 2026 as a new CPU benchmark — Prof_DavidBader · 2026-07-22
- Hybrid and local inference are emerging as a response to AI energy and token costs — dmitry140 · 2026-07-22
- NVIDIA details Vera CPU with 2x performance claims and a 22,000-core rack — ryanshrout · 2026-07-22
- NVIDIA says Vera Rubin NVL72 delivers 10x more tokens per megawatt than Blackwell — nvidia · 2026-07-22