Soft-MTP: a one-line fix removes noisy gradients from multi-token prediction
pmttyji · reddit · 2026-10-30
New paper identifies a flaw in standard MTP training: the MTP cross-entropy gradient decomposes into a useful KL self-distillation term plus a low-SNR residual that interferes with learning at scale. Soft-MTP uses a KL gradient-recovery trick via straight-through gradient routing to keep only the KL signal — a one-line code change, no extra parameters or overhead, compatible with DeepSeek- and Meta-style MTP. From 167M to 1.2B params it delivers lower NTP perplexity, faster speculative decoding, higher zero-shot accuracy, and stronger math/code reasoning after finetuning.
Related event: Soft-MTP Fixes Noisy Multi-Token Prediction Gradients for Free Gains(2 posts)→
More from Research
- Eilenberg's visionary 'Automata, Languages and Machines' gets a fresh recommendation — abhi9u · 2026-10-30
- Learning RL from scratch: one-page notes site de-jargonizes reinforcement learning basics — East-Muffin-6472 · 2026-10-30
- "Your LLM evaluation is lying to you": temp-0 Qwen3-235B gives 80 different answers in 1000 runs — irombie · 2026-10-30
- inclusionAI open-sources ZwZ2 vision models: +4.99 points across 25 benchmarks without zoom tools — jacek2023 · 2026-10-30
- Epoch data: leading chips get ~40% more efficient yearly, but AI power draw keeps climbing — teortaxesTex · 2026-10-30
- Vicki Boykis on ONNX inference for retrieval and ranking: slides from DataPhilly talk — vboykis · 2026-10-30