The 'Stochastic Parrot' Debate: Scholars Clash Over LLM Characterizations

Episode 1 · Researchers Debate Whether "LLMs Are Just Next-Token Predictors" Is a Meaningless Claim (2026-09-25, 7 posts)

On Sep 25–26, researchers clashed on Twitter/Bluesky over the popular claim that "LLMs are just next-token predictors / stochastic parrots." Aaroth opened by arguing the phrase is contentless: any mapping from input to output strings can be decomposed into a series of next-token predictions, so the claim conveys nothing substantive about LLMs. The debate then extended to whether human brains are isomorphic to LLMs.

Confirmed

• Aaroth posted on Sep 25, adding historical context: ten years ago, whether representing output distributions as next-token distributions posed computational difficulties was a genuinely interesting question, but practice showed the worry never materialized.
• Princeton's Aran Nayebi relayed the debate on Sep 26, noting participants including MIT's Andrew Ilyas and Aaron Roth; Aaron Roth similarly argued the "next-token prediction" framing lacks content.
• A second thread pitted Aaroth against @10001plateaus over brain mechanisms. @10001plateaus countered that saying "the brain picks the next word by producing ML scores" is flatly wrong: brains can think non-sequentially, and they have a mechanism for choosing "what to say" unlike an LLM's.
• Aaroth responded that any probability distribution can be written as a product of element-wise conditional distributions (exactly the LLM output form), yet this does not mean the producing system is inherently sequential; one cannot infer cognitive limits from a representational choice.

Why it matters

• The debate highlights a common misjudgment of LLM capability boundaries: conflating output form (token-by-token generation) with internal mechanism or reasoning capacity (inability to think non-sequentially), an error made by both LLM detractors and hypesters.
• It also links two threads: the decade-old learnability question about next-token representation has been settled by practice, while whether the human brain is a "prediction machine" remains open.

• 'Next-token predictor' is a contentless way to describe LLMs — Aaroth · 2026-09-25
• The decade-old worry that next-token representation would hinder learning turned out unfounded — Aaroth · 2026-09-25
• Why 'LLMs are just next-token predictors' is a contentless argument — Aaroth · 2026-09-25
• Researchers call 'next token predictor' a contentless critique of LLMs — aran_nayebi · 2026-09-26
• "Brains aren't next-token predictors": the counterpoint in an LLM-vs-brain debate — 10001plateaus · 2026-09-26
• Debate on X: Is a brain a next-token predictor? It's just math, argues one side — Aaroth · 2026-09-26
• LLM output distributions being factored doesn't mean they can't think out of order — Aaroth · 2026-09-26

Episode 2 · Melanie Mitchell Sparks Debate: Today's Systems Are Not LLMs (2026-09-27, 31 posts)

Starting September 27, Santa Fe Institute professor Melanie Mitchell argued on X that "what we have now are not LLMs," igniting a sprawling debate over the Stochastic Parrots paper and the definition of "LLM." Her core claim: a language model proper captures the statistical structure of language (n-grams, Elman networks, GPT-2), whereas today's systems undergo heavy post-training and embed many external software components, so attacking them with "stochastic parrots" is a straw man—the paper targeted 2021-era LLMs. As of publication, the dispute remains an exchange of views with no resolution.

Confirmed

• Mitchell clashed with Boaz Barak (University of Chicago) over definitions: Barak asked whether "LLM" and "stochastic parrot" apply only to pre-InstructGPT models, noting people have called the newer systems LLMs all along; Mitchell reiterated her view of the language-model lineage and said she only wants to correct the term's misuse—today it denotes any large AI system containing a pretrained language model, not its original meaning in the paper.
• mmitchellai clarified the paper's scope: "LLM" there meant "pretrained," "unsupervised" language models, reflecting the authors' mental model at writing time; they also noted having co-written with Emily Bender a piece explaining how "stochastic parrot" refers to LLMs.
• Google DeepMind's Andrew Lampinen posted a long thread countering Bender & Koller's core technical claim—that systems trained only on language learn mere form and no grounding-dependent meaning—arguing meaning can be learned from language alone and calling "stochastic parrot" itself "much ado about nothing," ironically mirroring the authors' own misunderstanding of language.
• Redwood Research's Blanche Minerva listed a five-step pipeline (pretrain on text, teach tool-calling tokens, teach the command line, etc.) to refute "LLMs are just next-token predictors," and asked Mitchell whether a transformer with command-line access that runs programs but does no input/output processing counts as a stochastic parrot—and whether the criterion depends on training method.
• Further technical pushback: francip argued pretraining, SFT, preference optimization and RL all modify the same weights and the same inference process yielding a token conditional distribution, so post-training alone cannot demarcate "language model"; DeepMind's Nathaniel Daw added that instruction tuning and RLHF shifting behavior away from pure next-token prediction has been true since ChatGPT. lusichu rebutted that Mitchell misread the paper's meaning.
• The paper's authors responded: Timnit Gebru said she would no longer engage with people obsessed with the argument, asking "why does an accurate description of LLMs upset you so much"; cloneofsimo quote-posted with an image.
• Others backed Mitchell: Brad Buchsbaum said the paper "ornithomorphizes" LLMs; tallinzen—rarely on the paper's side—argued word meanings shift over time, and judging the 2021 paper by today's systems is anachronistic; Oxford's Michael Bronstein said the classic critique, even with generous discounts such as stripping away surrounding software, should be retired. @aidanclark separately mocked imagegen failures as evidence current models are "dumb as a brick," far from ASI.

Unconfirmed

• No consensus exists on the boundary of "LLM": whether post-RLHF, tool-using systems still count as language models, and whether the "stochastic parrot" critique is thereby invalidated, remain unresolved points of contention.

Why it matters

• The dispute touches on whether one of AI's most-cited critical frameworks still applies: if today's systems differ in composition and capability from 2021's LLMs, safety and ethics discussions built on "stochastic parrots" may need to recalibrate their terminology and targets.

• Researcher pushes back on 'stochastic parrot' framing: even pure LLMs no longer fit it — BlancheMinerva · 2026-09-27
• Mitchell points to his and Emily Bender's essays on how "stochastic parrot" refers to LLMs — mmitchell_ai · 2026-09-27
• Melanie Mitchell: 'Stochastic parrots' is a strawman against RL-post-trained models — MelMitchell1 · 2026-09-27
• Blanche Minerva's five-step pipeline shows why "just next-token prediction" no longer fits LLMs — BlancheMinerva · 2026-09-27
• Researcher asks: a transformer with a command line — still a 'stochastic parrot'? — BlancheMinerva · 2026-09-27
• Dev challenges "stochastic parrot" critics: a transformer with a shell — how many lines until it's one? — BlancheMinerva · 2026-09-27
• A meme answers Timnit Gebru's 'stochastic parrots' defiance — cloneofsimo · 2026-09-27
• Mitchell: stochastic parrot critique targeted 2021 LLMs, not today's AI systems — PMinervini · 2026-09-27
• Mel Mitchell argues today's AI systems are no longer LLMs, term misuse muddies debate — _aidan_clark_ · 2026-09-27
• User roasts imagegen being 'dumb as bricks' as evidence we're nowhere near ASI — _aidan_clark_ · 2026-09-27
• Melanie Mitchell backs claim that today's systems aren't really LLMs, citing original definition — MelMitchell1 · 2026-09-27
• Melanie Mitchell vs DeepMind Co-founder: Do Today's Models Still Count as 'Stochastic Parrots'? — _aidan_clark_ · 2026-09-27
• Boaz Barak pushes back on Mitchell: were 'stochastic parrots' only pre-2022 models? — boazbaraktcs · 2026-09-27
• Melanie Mitchell vs. Boaz Barak: does 'stochastic parrots' still apply to today's LLMs? — MelMitchell1 · 2026-09-27
• Melanie Mitchell: language models span n-grams to GPT-2, agentic systems are another matter — MelMitchell1 · 2026-09-27
• Melanie Mitchell argues RLHF-trained models are no longer true language models — MelMitchell1 · 2026-09-27
• Melanie Mitchell: Post-Training Like RLHF Makes Models No Longer Statistical Language Models — MelMitchell1 · 2026-09-27
• Debate: raw LLMs without any harness already refute the 'stochastic parrot' claim — Zergylord · 2026-09-27
• Ex-Google Research VP Aidan Clark: "What We Have Now Are Not LLMs" — giffmana · 2026-09-27
• Google researcher Lampinen pens long thread rebutting the stochastic parrots argument on LLM meaning — AndrewLampinen · 2026-09-27

11 more related posts →