Neuroscientist pushes back on Goodfire's interpretability optimism, citing polysemanticity in brains
patrickmineault · x · 2026-10-18
nickcammarata argues mechanistic interpretability may be the most important scientific problem today—alignment is impossible without knowing what we're building. Early large-scale automated interpretability work at OpenAI left him pessimistic when only 2% of neurons were legible with primitive tools; with modern agentic tools, the vast majority of neurons in models up to trillions of parameters appear legible.
Neuroscientist HistedLab offers a counterpoint from brains: polysemanticity and distributed representations are widely seen in biological brains—a theme central to his career—cautioning against pure optimism on the AI side.
More from AGI Musings
- Mathematician: Pages of Routine Analysis Work Well, and AI Makes Writing Them Easier — jessi_cata · 2026-10-18
- Ezra Klein runs six straight AI podcast episodes, pushing back on left's AI-skepticism — Miles_Brundage · 2026-10-18
- Anthropic staff's answers imply 17.1% GDP growth in 2030 vs FOMC's 2% — soumitrashukla9 · 2026-10-18
- Sentdex: models arrive faster than I can test them — home agent swarms are coming — Sentdex · 2026-10-18
- Reddit essay argues there is no logical basis to claim AI cannot feel — s0me__dude · 2026-10-18
- Andrew Ng: 'AI for Science' will become meaningless as AI becomes every scientist's tool — andrewgwils · 2026-10-18