MIT's Soft-MTP Fixes Noisy Gradients in Multi-Token Prediction

MIT's paper introduces Soft-MTP, a one-line fix that removes about 80% of noisy gradients in standard multi-token prediction training, improving LLM pretraining for free.

2026-11-03 ~ 2026-11-04 · 2 related posts

• MIT's Soft-MTP: one-line fix removes 80% noisy gradient, boosts MTP pre-training for free — kastnerkyle · 2026-11-03
• Soft-MTP strips noisy gradients from multi-token prediction training at no extra cost — burkov · 2026-11-04