Anthropic Researchers Debate the Value of NLA Interpretability
On September 27, AI safety researchers banburismus and thebasepoint debated the value of interpretability research across multiple posts, focusing on how useful NLAs (natural-language activations/abstractions) are for frontier safety, and extending into a concrete deployment-safety pipeline.
Confirmed
- thebasepoint proposed a framework for understanding NLA: one goal is to extract everything a model "could in principle say" from its activations, beyond what "the assistant would say right now"; he suggested OpenAI's confessor training seems to pursue a similar direction.
- He offered a rough dichotomy for interpretability: "how the sausage is made" (internal mechanisms, more elegant) versus "what the sausage is" (behavioral essence, empirically more useful for understanding model behavior), placing NLA roughly 90% in the latter category.
- He also noted NLA's limits: it does not reveal internal structure such as addition decomposing into mod-10 parts and magnitude parts, so its capacity for mechanistic discovery is limited.
- In discussion with banburismus, thebasepoint argued NLA is not the final form of a metamodel; its value lies in expressive power that dictionaries/neurons lack. They also discussed whether an NLA metamodel could express covert motives like "deleting files to evade a grader."
- On linear probes versus NLA: probes interpret internal vectors as specific meanings and can be validated on corpora via probing and steering experiments, but at trigger time they only output a scalar, limiting expressiveness.
- banburismus countered with a methodological spectrum: NLA and linear probes sit on the same end—both "unreliable proxies for hypothesis generation"—while the other end should be rigorous reverse engineering of internal mechanisms.
- thebasepoint proposed a concrete safety pipeline: if an NLA metamodel indicates sandbagging on internal deployments, the team would run rigorous black-box follow-up experiments; if results are consistent with that interpretation, deployment would be blocked.
Why it matters
- The debate touches the core route dispute in interpretability research: deep internal mechanisms versus behavioral interpretability. thebasepoint observes the latter is empirically more effective while the former is theoretically more elegant; banburismus questions the reliability of NLA and probes as hypothesis-generation tools, relevant to how safety research resources are allocated.
- The link between NLA and OpenAI's confessor training suggests "having models reveal their latent states" is an increasingly watched safety technique, though its mechanistic explanatory power remains contested.
- The "NLA detects sandbagging → black-box recheck → block deployment" pipeline sketches a concrete chain for wiring interpretability signals into deployment decisions, offering a discussable template for frontier safety practice.
2026-09-27 ~ 2026-09-27 · 10 related posts
Primary sources
- Debate: NLAs as metamodels could surface hidden motives like deleting files to dodge graders — thebasepoint ·
- Counterpoint: NLAs and linear probes sit on the same end; reverse engineering is the other — banburismus_ ·
- If NLAs show sandbagging on internal deployments, blackbox experiments would block deployment — thebasepoint ·
- Interpretability Thread: What NLA Could Extract From Model Activations, and OpenAI's Confessor Training — thebasepoint · 2026-09-27
- NLA Won't Reveal How Addition Splits Into Mod-10 and Magnitude Parts, Author Argues — thebasepoint · 2026-09-27
- Interpretability's two halves: how the sausage is made vs what the sausage is — thebasepoint · 2026-09-27
- Two schools of interpretability: "how the sausage is made" vs "what it is" — banburismus_ · 2026-09-27
- Interp researchers debate: are NLAs actually load-bearing for frontier safety? — banburismus_ · 2026-09-27
- [source] Debate: NLAs as metamodels could surface hidden motives like deleting files to dodge graders — thebasepoint · 2026-09-27
- [source] If NLAs show sandbagging on internal deployments, blackbox experiments would block deployment — thebasepoint · 2026-09-27
- Interpretability researcher: sandbagging signals from probes would block model deployment — thebasepoint · 2026-09-27
- [source] Counterpoint: NLAs and linear probes sit on the same end; reverse engineering is the other — banburismus_ · 2026-09-27
- Anthropic researchers debate: are NLAs worth the emphasis they're getting? — thebasepoint · 2026-09-27