LLMs Aren't Markov Chains: Dev Pushes Back on "Next-Word Parrot" Claim
austinc3301 · x · 2026-10-04
Responding to the claim that "AI works like human minds scanning sentences and picking the most common next word," austinc3301 laid out a full rebuttal:
- The confusion stems from conflating the training task (next-word prediction) with the trained model, which does far more sophisticated things;
- Modern LLMs have extremely complex world models and mix heuristics with algorithms for different problems — a Markov chain predictor is not how they choose the next token;
- He adds that when people say AI resembles human minds, they mean higher-level emergent behavior: natural language communication, theory of mind, and modeling human values well.
The exchange highlights a common AI-literacy misconception: using the training objective to dismiss inference-time capabilities.
Related event: Developers push back on claim that AI is just next-word prediction(2 posts)→
More from AGI Musings
- Four roles for humans in an AI world: from human premium to augmented craft — msharmas · 2026-10-04
- Bindu Reddy bets on human labs over superintelligence, expects open-source surprises — bindureddy · 2026-10-04
- Is functionalism about Claude circular? A substrate debate on X — ctjlewis · 2026-10-04
- Brain FLOP/s estimates may bound coding compute but not consciousness — JoshPurtell · 2026-10-04
- Bounding AGI timelines: Josh Purtell on Carlsmith-style Fermi estimates — JoshPurtell · 2026-10-04
- AI writes 20-30% of Microsoft's code: educators debate what schools should still teach — Stefania_druga · 2026-10-04