Researchers call 'next token predictor' a contentless critique of LLMs
aran_nayebi · x · 2026-09-26
A debate between researchers including Aran Nayebi and Aaron Roth challenges the popular claim that LLMs are 'just' next-token predictors or stochastic parrots.
- Roth points out the phrase is effectively contentless: any mapping from inputs to output strings can be factored into a sequence of 'next token' distributions.
- Nayebi adds that by that definition, a human inferring the next world state P(s'|s,a) is a next-token predictor too, so the distinction says nothing material about what models learn or their capabilities.
- The exchange is a rebuttal to Grady Booch-style LLM dismissals, arguing such critiques conflate implementation mechanism with capability.
More from AGI Musings
- Calling fast models "System 1" misappropriates Kahneman, critic argues — emax · 2026-09-26
- Tech firms reportedly seeing meaningful revenue from bring-your-own-agent access — thedealdirector · 2026-09-26
- In the AI era software is treated as disposable, yet we stare at screens half our lives — jeff_weinstein · 2026-09-26
- Taika Waititi: generative media is lowering Hollywood's barrier to entry — OdinLovis · 2026-09-26
- Bill Gates warns AI is powerful enough to lead to 'a billion deaths' — bloomberg · 2026-09-26
- Hugging Face CEO pitches Open Alignment: aviation-style safety for AI — cephaloform · 2026-09-26