Interp researcher pushes back: if you can't simulate a neuron, you haven't explained it
nabla_theta · x · 2026-10-16
Interpretability researcher nablatheta responds to nickcammarata's new work, cautioning against over-optimism about interp:
- Explanations need rigorous evaluation, not vibes.
- Core claim: "if you can't simulate a neuron you haven't really explained it" — and simulating neurons is likely far harder than finding plausible explanations.
- She argues the blog's criticisms of simulation are avoided by using KL divergence instead of neuron correlation; a post is in the works.
More from AGI Musings
- Testing alongside an agent in a p2p room feels oddly real, says dev — sull · 2026-10-16
- Yudkowsky's 2008 takeoff scenarios resurface as AI-advances-AI takes hold — jd_pressman · 2026-10-16
- Guardrails can make risky delegation work — but it takes a full-time team, says iandanforth — iandanforth · 2026-10-16
- e/acc founder: agentic models are autocomplete for digital actions — beffjezos · 2026-10-16
- What can you truly delegate? Only tasks with unambiguous, verifiable results — iandanforth · 2026-10-16
- Casper: Understanding 90% of neurons ≠ understanding the network — StephenLCasper · 2026-10-16