Interpretability solved—then what? Najoung Kim proposes filtering for recurrent constructs
najoungkim · x · 2026-11-19
- Thought experiment: assume interpretability is solved—we can faithfully identify the representations and computations behind any capacity in any network. We'd then obtain a flood of "machine constructs," not all scientifically meaningful.
- Organizing dimensions: theoretical status (accepted vs novel), domain status (linguistic vs non-linguistic), ontological status (privileged vs not), and simplicity.
- Core claim—recurrent constructs: constructs shared by models diverging across specification axes (architecture, data, objective) yet converging on behaviors of interest are the most informative; since interpretability targets are not models of people, recurrence despite specification divergence best strengthens inductive inference to the human mind.
- Positioning: this is a fundamentally different, complementary mode of discovery versus computational modeling; current empirical practices make it hard to apply, and the authors propose ways forward.
More from AGI Musings
- AI governance scholar Luiza Jarovsky calls for a global ban on RSI — LuizaJarovsky · 2026-11-19
- Stanford AI100 Releases Third Report on the Scaling Era and Rise of Generative AI — StanfordHAI · 2026-11-19
- Elicit on agent swarms: pharma winners of 2036 will robustly deploy them without going rogue — jungofthewon · 2026-11-19
- Pro vs. con arguments on AI "handover" rest on different moral foundations — 1a3orn · 2026-11-19
- Yoav Goldberg: server-side session APIs will make LLMs opaque — as bad as closed weights — yoavgo · 2026-11-19
- Google Brain researcher warns LLM literature audits lack judgment, risk witch hunts — rbhar90 · 2026-11-19