Researchers call 'next token predictor' a contentless critique of LLMs
aran_nayebi · x · 2026-09-26
A debate between researchers including Aran Nayebi and Aaron Roth challenges the popular claim that LLMs are 'just' next-token predictors or stochastic parrots.
- Roth points out the phrase is effectively contentless: any mapping from inputs to output strings can be factored into a sequence of 'next token' distributions.
- Nayebi adds that by that definition, a human inferring the next world state P(s'|s,a) is a next-token predictor too, so the distinction says nothing material about what models learn or their capabilities.
- The exchange is a rebuttal to Grady Booch-style LLM dismissals, arguing such critiques conflate implementation mechanism with capability.
More from AGI Musings
- An AI model's 8-point answer on how humans can thrive in the 'Age of Miracles' — daniel_mac8 · 2026-11-11
- Podcast with X's biggest AI account: how one post made Manus famous in the West — kimmonismus · 2026-11-11
- Area Chair: reviewing AI-written reviews of AI-generated papers, the only human in the loop — murat_kocaoglu_ · 2026-11-11
- Dev Dimillian calls for content on AI-assisted work vs. pure AI slop — Dimillian · 2026-11-11
- How long until a phone-runnable model solves a Millennium Prize Problem — and Siri still lags? — paul_cal · 2026-11-11
- Superforecasters cut ASI timeline by 3 years to 2028, citing METR report — ben_j_todd · 2026-11-11