Ex-OpenAI interpretability lead: neurons in trillion-param models are mostly legible
adityaag · x · 2026-10-16
Nick Cammarata, co-author of early mechanistic interpretability work at OpenAI who later led its first large-scale automated interpretability project, argues interpretability may be the most important scientific problem today — alignment is unsolvable without knowing what we're building.
He left OpenAI pessimistic: with primitive tools, only 2% of neurons were legible and networks looked like tangled knots. With modern agentic tools and closer inspection, the picture flipped — the vast majority of neurons and circuits are legible even in models up to trillions of parameters. Reposter zoink calls it possibly the most important essay of 2026, framing it as choosing optimism and building tools to face the challenges head-on.
More from AGI Musings
- Beff Jezos declares 'civilizational inflection point' in the acceleration of technological progress — beffjezos · 2026-10-16
- Data scientist: pivoting to AI engineering is the safest career bet for the next 5 years — mdancho84 · 2026-10-16
- AI will be superhuman at science regardless, and Europe can't opt out — teortaxesTex · 2026-10-16
- beffjezos to Aaron Levie: techno-optimism is hyperstitious — basedjensen · 2026-10-16
- AI demos haven't changed since 2023, only gotten better — same scenarios — yangyi · 2026-10-16
- CFR experts: the autonomous weapons debate still has a definition problem — mchorowitz · 2026-10-16