Dietterich: pretraining's next-token task already forces models to abstract and generalize
tdietterich · x · 2026-09-25
Pushing back on the "pretraining is just parroting" framing, AI researcher Thomas Dietterich argues the next-token task is demanding enough to force models to abstract and generalize in useful ways even before fine-tuning. SFT and RL then go beyond predicting language to predicting correct multi-token answers; pretraining mainly builds good internal representations plus a stochastic generator of candidate solutions.
Related event: Dietterich: LLMs Go Beyond Next-Token Prediction(2 posts)→
More from Research
- Tuva Labs takes 1st on ARC-AGI-2, 2nd on ARC-AGI-3 in ARC Prize 2026 — GregKamradt · 2026-11-11
- Sakana AI × Pathway Post-Transformers lecture on AI and the brain now online — kaixhin · 2026-11-11
- Defending METR: public data can't second-guess its AI speedup findings — peterwildeford · 2026-11-11
- Adversarial RL envs for backdoor insertion vs. detection: where does it converge? — DimitrisPapail · 2026-11-11
- DiffusionEngine: an open-source modular PyTorch DDPM framework seeking contributors — Euphoric-Incident-93 · 2026-11-11
- UniPath (NeurIPS 2026): adaptively coordinating understanding and generation in one model — jindong_wang92 · 2026-11-11