MIT's Soft-MTP: one-line fix removes 80% noisy gradient, boosts MTP pre-training for free
kastnerkyle · x · 2026-11-03
An MIT paper ("Multi-Token Prediction Done Right") identifies a flaw in standard Multi-Token Prediction training: 80% of the MTP gradient comes from a low signal-to-noise residual that interferes with both next-token and multi-token learning, increasingly at scale.
The method
- The MTP cross-entropy gradient decomposes exactly into a KL self-distillation term pulling each MTP head toward the model's own NTP distribution (improving speculative decoding), plus a noisy residual.
- A KL gradient-recovery trick recovers the KL gradient exactly by combining cross-entropy gradients via straight-through routing when distributions share a linear output head — no vocabulary-sized logit matrices, compatible with fused cross-entropy kernels.
- Soft-MTP is a one-line code change, adds no parameters, no measurable overhead, and supports both DeepSeek- and Meta-style MTP.
Results
- From 167M to 1.2B parameters: lower NTP perplexity, faster speculative decoding, higher zero-shot accuracy at every scale.
- At 1.2B, HumanEval Pass@64 improves from 40.85% to 43.90%, with stronger math and code reasoning after fine-tuning.
- The authors call it "a free lunch."
More from Research
- mlx-learn ships scikit-learn API on Apple Silicon with up to 300x faster inference — HankYeomans · 2026-11-03
- CMU launches first Protostar Challenge on climate risk AI, open until Nov 13 — evijit · 2026-11-03
- Tivadar Danka publishes a complete math roadmap for machine learning: linear algebra, calculus, probability — TivadarDanka · 2026-11-03
- Formalizing CFSG Uncovers 96 Math Errors in 47 Peer-Reviewed Papers, Including Annals of Mathematics — skdh · 2026-11-03
- SEC-Bench Pro: frontier labs test agents on 183 real V8 and SpiderMonkey vulnerabilities — NielsRogge · 2026-11-03
- PhD thesis from Aengus Lynch shows aligned AI stays vulnerable: 89% jailbreak, 96% blackmail — StephenLCasper · 2026-11-03