Early OpenAI interpretability researcher: most neurons in trillion-param models are now legible
CatAstro_Piyush · x · 2026-10-31
Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI, argues interpretability may be the most important scientific problem today since alignment is impossible without knowing what we're building. His early large-scale automated interpretability project could make only 2% of neurons legible, but with modern agentic tools the vast majority of neurons and circuits — even in models up to trillions of parameters — are now readable. The quoted retweet includes a no-prereqs visual explainer of the field's historical context.
More from AGI Musings
- Epoch index shows open-weight models closing the gap with closed frontier to under 10 points — SpencrGreenberg · 2026-10-31
- Metaculus forecasters put 96% odds on open-weight AI reaching superhuman hacking by July 2027 — SpencrGreenberg · 2026-10-31
- Anti-AI art bias is privileged gatekeeping, argues X user — heyneighbor · 2026-10-31
- UChicago study: students demand thousands to give up smartphones as screens kill street play — aakashgupta · 2026-10-31
- TeortaxesTex: a DeepSeek paper on building unrestricted models would be more disruptive than the models — teortaxesTex · 2026-10-31
- Bindu Reddy: Firms treat AI as a $20/month perk, so the $1T valuations can't hold — bindureddy · 2026-10-31