Early OpenAI interpretability researcher: most neurons in trillion-param models are now legible
typewriters · x · 2026-10-16
An early mechanistic interpretability researcher at OpenAI (nickcammarata) posted a long thread, shared by Goodfire AI board members on announcement day:
- Interpretability "might be the most important scientific problem in the world" — alignment is unsolvable without knowing what we're building
- His early OpenAI work left him pessimistic: networks looked like arbitrary tangled knots, with only 2% of neurons legible under primitive tools
- With modern agentic tools and closer inspection, the picture has flipped: in models up to trillions of parameters, the vast majority of neurons and their circuits are legible
More from Research
- UCSD Researchers Propose Dataset Watermarking Detectable via Black-Box API Access — yuxiangw_cs · 2026-10-16
- EmbodiedEvo tops RoboDojo as agents inherit reusable skills and lessons — Aiden_Tech_Ai · 2026-10-16
- CHI reviewer laments: papers' Discussion sections now cite nothing — IanArawjo · 2026-10-16
- Casper: Understanding 90% of neurons ≠ understanding the network — StephenLCasper · 2026-10-16
- NVIDIA Launches cuPotential in CUDA-X to Bring AI Agents to Semiconductor Materials Simulation — tomaszbednarz · 2026-10-16
- Agentic Search Systems Are Rewarded Only for Final Answers, Raising Faithfulness Concerns — burkov · 2026-10-16