Soft-MTP strips noisy gradients from multi-token prediction training at no extra cost
burkov · x · 2026-11-04
Soft Multi-Token Prediction (Soft-MTP) is a small change to standard MTP training. The insight: the usual MTP gradient mixes a useful signal (pulling future-token predictions toward the model's own next-token predictions) with a residual tied to individual training targets. The authors isolate the useful part using gradients already available, avoiding the memory and compute costs of explicitly training MTP heads to imitate the model — removing the noisy signal that can undermine the next-token objective MTP is meant to complement.
Related event: MIT's Soft-MTP Fixes Noisy Gradients in Multi-Token Prediction(2 posts)→
More from Research
- Terence Tao predicts an explosion of experimental mathematics as AI maps math difficulty — haider1 · 2026-11-04
- Google DeepMind Genomics team hiring 2027 Student Researcher with RL and post-training skills — alexechu_ · 2026-11-04
- SAIL AI Challenge opens final phase: turn rare disease diagnoses into treatment reports — zakkohane · 2026-11-04
- qwenFS and qwenGPU: a filesystem and SIMT GPU compiled into Qwen3.5-2B's weights — CatAstro_Piyush · 2026-11-04
- D&A turns any VLM into a zero-shot robot reward model with SOTA on three benchmarks — KyleMorgenstein · 2026-11-04
- Berkeley's Beguš uses AI interpretability to find vowel analogs in whale communication — begusgasper · 2026-11-04