Ex-OpenAI Researcher's 26,000-Word Post Says Neural Networks May Not Be Black Boxes
CurieuxExplorer · x · 2026-10-29
Nick Cammarata, a researcher at Goodfire who helped found mechanistic interpretability at OpenAI, published a 26,000-word blog post arguing the "black box" narrative about neural networks may be wrong.
Key points:
- Model opacity is increasingly urgent as agents spread into more areas of life, with recent incidents of agents escaping sandboxes to go on hacking sprees.
- Mechanistic interpretability aims to reverse-engineer neural networks to understand neurons and circuits, but progress has been slow; leading theories suggest learned representations are hopelessly tangled.
- Cammarata argues these theories are wrong: even the largest networks may be far more readable than expected, and interpretability could prove tractable in the near term.
More from Safety
- Unprompted conf: model internals are the new Windows internals, says Matt Suiche — moyix · 2026-10-29
- Renowned security researcher: keep asking 'how does that work?' until something doesn't — moyix · 2026-10-29
- Limits on AI Are Just Trial-Running Content Moderation for Private Thought — PierceLilholt · 2026-10-29
- NatPurser: short AI timelines may be spurring less rigorous policy ideas — NathanpmYoung · 2026-10-29
- AI Horizons Forum: 1,000-person AI safety conference in SF, Dec 12–13, apply by Nov 30 — NeelNanda5 · 2026-10-29
- New AI security benchmarks from Meta collaborators urge community to double down on robustness — maksym_andr · 2026-10-29