The decade-old worry that next-token representation would hinder learning turned out unfounded
Aaroth · x · 2026-09-25
@Aaroth reflects on a key shift in how we think about LLMs: a decade ago it was a genuinely interesting computational question whether representing output distributions as next-token distributions would make it hard to learn useful behaviors. It didn't.
- Any text-outputting system can trivially be written as a "next token predictor" — a syntactic fact about probability distributions
- This says nothing about whether brains work similarly to LLMs internally
More from AGI Musings
- An AI model's 8-point answer on how humans can thrive in the 'Age of Miracles' — daniel_mac8 · 2026-11-11
- Podcast with X's biggest AI account: how one post made Manus famous in the West — kimmonismus · 2026-11-11
- Area Chair: reviewing AI-written reviews of AI-generated papers, the only human in the loop — murat_kocaoglu_ · 2026-11-11
- Dev Dimillian calls for content on AI-assisted work vs. pure AI slop — Dimillian · 2026-11-11
- How long until a phone-runnable model solves a Millennium Prize Problem — and Siri still lags? — paul_cal · 2026-11-11
- Superforecasters cut ASI timeline by 3 years to 2028, citing METR report — ben_j_todd · 2026-11-11