Neuron labels are like code comments: debate on interpretability vs CoT
jessi_cata · x · 2026-10-16
A debate sparked by Nick Cammarata's new site on monosemanticity in neural networks.
Roko Mijic argues that accurately labeling neurons (though the labels are "a bit vibed") is like writing comments in code — the comments can be accurate but don't constrain or explain the program's behavior.
jessicata responds: interpretability will likely make neuralese more transparent, partially fixing wrong or opaque chain-of-thought, but carries more compute overhead than CoT — better suited for post-hoc incident investigation than running on everything.
Related event: Roko Critiques Neuron Labeling, Defends Chain-of-Thought(2 posts)→
More from AGI Musings
- Testing alongside an agent in a p2p room feels oddly real, says dev — sull · 2026-10-16
- Yudkowsky's 2008 takeoff scenarios resurface as AI-advances-AI takes hold — jd_pressman · 2026-10-16
- Guardrails can make risky delegation work — but it takes a full-time team, says iandanforth — iandanforth · 2026-10-16
- e/acc founder: agentic models are autocomplete for digital actions — beffjezos · 2026-10-16
- What can you truly delegate? Only tasks with unambiguous, verifiable results — iandanforth · 2026-10-16
- Casper: Understanding 90% of neurons ≠ understanding the network — StephenLCasper · 2026-10-16