François Fleuret: 'Just Predicting the Next Token' Is the Wrong Level of Description
francoisfleuret · x · 2026-09-26
Meta researcher François Fleuret pushes back on the popular 'LLMs just predict the next token' dismissal in a tweetstorm: it's as informative as describing a chess program as 'just moving electrons around' — technically true but the wrong level of abstraction.
The model does sample token by token, but according to an extremely complex distribution over sentences, and that's the meaningful level at which to understand what it does.
More from AGI Musings
- AI Is Making You Dumber: The Hidden Cost of Vibe Coding and Skill Atrophy — michalmalewicz · 2026-09-26
- If open source wins, AI value accrues to whoever serves inference cheapest — markjeffrey · 2026-09-26
- EU's top AI adviser: stop chasing US and China frontier models, fix productivity instead — alexvoica · 2026-09-26
- At OpenAI's 2026 AGI Economy Conference, researchers call for economic-task evals — soumitrashukla9 · 2026-09-26
- AI could codify tacit organizational knowledge, worth up to one US GDP, paper says — soumitrashukla9 · 2026-09-26
- Blaise Agüera y Arcas marks one year of 'What Is Intelligence?' — blaiseaguera · 2026-09-26