"The system has no self": critic says behavioral-explanation training misunderstands stateless LLM inference
gerardsans · x · 2026-09-05
Anthropic researcher akarvonen shared that training models on thousands of explanations of their own in-the-wild behaviors (ignored requests, coding errors) generalizes to held-out evals from a single dataset.
Gerard Sans pushed back that this misrepresents inference: every transformer pass is atemporal and stateless, with no self or continuity between prompts or even autoregressive steps — asking a model to explain itself is just another pass. He cited the arXiv paper "Counterfactual Token Generation in Large Language Models" as evidence that LLMs cannot reason about tokens generated in the past.
More from AGI Musings
- Margaret Mitchell: click-driven AI misinformation is causing real harm — mmitchell_ai · 2026-10-21
- Mathematician Daniel Litt concedes his AI bet, but says AI won't solve math — ziv_ravid · 2026-10-21
- AI SaaS monetization is cooked: consumers small, enterprises pay $20/month — menhguin · 2026-10-21
- Tim Dettmers: Academia is about to have a renaissance — the best AI work of the next decade happens in university labs — StasBekman · 2026-10-21
- AI saves scientists 7 hours a week, but validation is the new bottleneck — AnneliesGamble · 2026-10-21
- UW's Amy X. Zhang brings CSCW theory to human-agent collaboration in Stanford talk — amyxzh · 2026-10-21