LightSpec: Open-Sourcing Universal Dynamic MTP Framework
机器之心 · wechat · 2026-07-16
The LightLLM team has open-sourced LightSpec, a universal dynamic MTP inference framework, aiming to evolve dynamic multi-token prediction from a model-specific trick into a universal inference infrastructure.\n\nIts core is a two-stage, training-free runtime planner: it first dynamically determines the verification budget for the current round, and then sets the draft budget for the next. Instead of relying on a separately trained confidence model, scheduling leverages naturally generated timing tables and EMA acceptance rates during inference for online decisions. This ensures compatibility with different draft models like DSpark and Eagle3.\n\nThe article also details the full system implementation, featuring runtime cost tables, online acceptance stats, dynamic input construction, and pipeline coordination to maximize CPU scheduling and GPU execution overlap. Experiments show that under Qwen3-14B, H200, and 256 concurrency, this dynamic approach boosts throughput across various workloads, with particularly notable advantages at larger MTP step sizes.
More from Infra
- Cloud release velocity scores: AWS 95, Azure 90, Google Cloud 82 — DavidLinthicum · 2026-10-27
- Selling a 5090 to scale local inference: one business weighs 5 hardware paths — Anarchaotic · 2026-10-27
- AWS tutorial: fleet-wide edge CV model lifecycle with SageMaker SDK v3, ~$0.07 per run — AWS ML Blog · 2026-10-27
- Qwen3.8-27B hits 40 tok/s on Jetson Thor with INT4 + MTP, accuracy beats FP16 — No-Dragonfly6246 · 2026-10-27
- Celesto launches CI runners nearly 90% cheaper than GitHub Actions, with agent sandboxes — aniketmaurya · 2026-10-27
- Supabase founders on rebuilding the backend for a billion AI-built apps — Latent Space · 2026-10-27