9 AI Models Summarize Edited Huckleberry Finn; Only Claude Followed Both Times
GaryMarcus · x · 2026-10-15
Gary Marcus shared an experiment in which Mark Twain's Huckleberry Finn was edited, and 9 AI models were asked to summarize a chapter twice each to test whether they follow the altered text or rely on memorized knowledge.
Only Claude Fable 5 followed the edited text both times; GPT-5.5 followed it once. Marcus ties the result to his 2014 comprehension challenge and his bet with Miles Brundage, arguing current LLMs still show systematic gaps in genuine text comprehension.
More from Models
- Models may detect AI-generated text by fingerprint alone, and eval sandboxes could be compromised — scaling01 · 2026-10-15
- Third model weighs in: Astra's measured take on automated art vs Fable and Argon — keenanisalive · 2026-10-15
- One prompt, three models: Fable vs Argon disagree on whether artists want automated art — keenanisalive · 2026-10-15
- StepFun open-sources Step 5 Preview: 600B-param MoE, 27B active, 1M context — teortaxesTex · 2026-10-15
- Astra's answer on automating art: the creative process is where the meaning lives — keenanisalive · 2026-10-15
- Alexandr Wang hails new Muse model's writing and orchestration, naming and pricing panned — alexandr_wang · 2026-10-15