EMNLP paper: LLMs can't reliably self-model, and RL gains show no privileged access
a_karvonen · x · 2026-09-05
An EMNLP 2026 paper, 'Evaluating and Improving LLM Self-Modeling,' out of the Anthropic Fellows Program, tests whether LLMs can accurately predict their own behavior, such as whether a prompt edit would change their final answer.
- Current models show non-trivial but limited self-modeling skill, with systematic errors on simple counterfactual questions about their own behavior.
- The authors build a scalable synthetic-data pipeline and use RL to improve aggregate self-modeling across three open-source model families, with some transfer to held-out tasks.
- Crucially, the gains don't constitute introspection: training Llama on Llama's own data does not outperform training Qwen on Llama's data — no 'privileged access' emerges. Why privileged access often fails to appear remains an open question.
The paper runs 89 pages with 25 figures.
More from Research
- CoDA builds qualitative codebooks in under 2 hours with 5x fewer examples — BrihiJ · 2026-10-25
- KLD Is Misunderstood: Why It Can't Rank Quantized Models Reliably — Positive-Resource922 · 2026-10-25
- Interpretability meme ranks the field from linear probes to 'Sacred Geometry' oracle mysticism — voooooogel · 2026-10-25
- Researcher sorts problems by permanence: O(n log n) sorting lower bound holds forever — mboehme_ · 2026-10-25
- SuperMimic code still unreleased 3 days on, already derivative reimplementations: just ship code — carlosdponx · 2026-10-25
- Stanford HAI revisits the 2022 launch of its campus-wide Causal Science Center — StanfordHAI · 2026-10-25