Roko: neuron labels are like code comments — preserving CoT matters far more
RokoMijic · x · 2026-10-16
Roko Mijic critiques Nick Cammarata's buzzed-about monosemanticity work: labeling neurons is like writing code comments — sometimes accurate, not always. Controlled-setting labels predict "climate," but the real world delivers "weather." If AI risk is fat-tailed, as Doomers contend, a climate-only tool is useless. Chain of thought is the cheap real-time monitor of neural "weather" — a short string, not a massive network. Preserving CoT matters more than all neuron-level interpretability, yet we're failing at it since no law bans its obfuscation and OpenAI started cheating.
Related event: Roko Critiques Neuron Labeling, Defends Chain-of-Thought(2 posts)→
More from AGI Musings
- Ben Affleck calls AI 'just matrix multiplication'; Grady Booch echoes the skepticism — Grady_Booch · 2026-10-16
- Everyone has a precise AGI definition—until a model gets close to meeting it — VraserX · 2026-10-16
- Viral take: If AI gives us time back, its greatest gift is revealing what we mistook for the cost of living — r0ck3t23 · 2026-10-16
- "AI automation = slop, AI augmentation = super intelligence" — the line CEOs should see — jacob_posel · 2026-10-16
- Reddit take: Building a successor AI species would itself be humanity's purpose fulfilled — Flashy-Disk311 · 2026-10-16
- When Simple Rules Beat AI: A Frozen Ride-Hailing State Machine at Dulles — shashib · 2026-10-16