Ex-OpenAI researcher claims most neurons are legible now; critics call it hype
anshulkundaje · x · 2026-10-16
- Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI, argues interpretability may be the most important scientific problem: alignment is impossible without knowing what we're building.
- He recalls pessimism from early work—primitive tools rendered only 2% of neurons legible—but says with modern agentic tooling, the vast majority of neurons in models up to trillions of parameters are now legible, with clear circuits.
- Stephen Casper pushes back: the essay (an opener for a forthcoming paper) echoes a recurring 'wrong abstractions, new approach will fix it' cycle, likely reviving activation atlas / polytope ideas with scaled automated analysis. He cites Goodfire's history of overhyping: 'hype until further notice.'
Related event: Claims of Interpretability Breakthrough Spark Fierce AI Safety Debate(4 posts)→
More from Research
- Free Textbook: Introduction to Machine Learning Covers the Math from Optimization to Generative Models — TheTuringPost · 2026-10-16
- Free ML math textbook: Laurent Younes' Introduction to Machine Learning covers Adam to generative models — TheTuringPost · 2026-10-16
- RFdiffusion3 published in Science, alongside Chaperone agent harness for bio-molecular design — anshulkundaje · 2026-10-16
- Microsoft paper: history-aware model routing lifts multi-turn agent task pass rate by 3.3 points — dair_ai · 2026-10-16
- Researcher: everyone should build a good eval at least once — bad evals are everywhere — gabriberton · 2026-10-16
- The square lattice connective constant remains unsolved — and might be ripe for AI — tak3sh8 · 2026-10-16