Soft-MTP Fixes Noisy Multi-Token Prediction Gradients for Free Gains

An MIT single-author paper proposes Soft-MTP, a one-line fix that corrects noisy gradients in standard multi-token prediction training, yielding free performance gains across scales and improved speculative decoding.

2026-10-30 ~ 2026-10-30 · 2 related posts

• MIT's Soft-MTP Fixes Noisy Gradients in Multi-Token Prediction, Free Perplexity and Reasoning Gains — Massachusetts-Institute-of-Technology · 2026-10-30
• Soft-MTP: a one-line fix removes noisy gradients from multi-token prediction — pmttyji · 2026-10-30