MIT's Soft-MTP Fixes Noisy Gradients in Multi-Token Prediction, Free Perplexity and Reasoning Gains
Massachusetts-Institute-of-Technology · hf · 2026-10-30
In a single-author MIT preprint, Jayden Teoh shows multi-token prediction leaks noisy negative gradients from the NTP loss into MTP heads via the shared LM head. Soft-MTP removes this residual with a straight-through KL recovery trick — h0[:, k:] += hk.detach() - hk — leaving h0 numerically unchanged and NTP gradients intact while recovering KL gradients. The fix improves perplexity, speculative decoding, and reasoning at no added cost.
More from Models
- Daily AI Brief: Google kills Jules into Antigravity, OpenAI doubles Pro usage limits — testingcatalog · 2026-10-30
- Researcher: concept neurons alone may not be enough to induce an AI persona — Sauers_ · 2026-10-30
- Non-coder ditches Codex and ChatGPT for Kimi Work, says the experience is simply nicer — doctor_dadbod · 2026-10-30
- Leaked 'DeepSeek-cyber' tease: open DeepSeek models near frontier on cybersecurity CVE benchmarks — teortaxesTex · 2026-10-30
- Anthropic's internal model found and exploited critical bugs in Chrome, Safari, Android and more — emmanuelvivier · 2026-10-30
- Deep Research with Gemini 4 Argon generates a full Amazon returns infographic — gaganghotra_ · 2026-10-30