Ex-OpenAI interpretability researcher: most neurons in trillion-parameter models are now legible
tszzl · x · 2026-10-26
- Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI, argues interpretability may be the most important scientific problem today: alignment is impossible without knowing what we're building.
- Early tools could only make 2% of neurons legible, which left him pessimistic—networks seemed like arbitrary tangled knots.
- With modern agentic tools, the picture flipped: the vast majority of neurons in models up to trillions of parameters are legible, with clear circuits.
- Quoted tweet from @saibojnal shows neurons are interpretable enough that models can modify their own weights toward desired traits (demo: 6 steps to turn a model into a "bee" persona; 4096 rewritten neurons persist after clearing context, trait shows 20% of the time on unseen questions). Blog post incoming.
More from AGI Musings
- AI Writing's Tell: Negating Claims Nobody Made — Illustrated by Hotel Scrambled Eggs — andersonbcdefg · 2026-10-26
- Dario Amodei warns AI uniquely suited to autocracy; critics call it a product showcase — max_paperclips · 2026-10-26
- 30% of US STEM PhD dissertations now contain AI-generated text, NBER paper finds — 0xkarasy · 2026-10-26
- None of the 11 grand challenges in single-cell science solved since 2020 — Sauers_ · 2026-10-26
- "All roads lead to the end of man": a grim AGI endgame thread — danfaggella · 2026-10-26
- Cybersecurity commentator warns AI-era cyberattacks could cripple global critical infrastructure — realsohamparekh · 2026-10-26