Neel Nanda Defends CoT Monitoring as Safety Tool; Critics Cite Faithfulness Gaps
AlexTensor · x · 2026-09-04
DeepMind interpretability researcher Neel Nanda pushed back on the take that Chain-of-Thought monitorability doesn't matter because interpretability will save us, calling CoT our best current safety tool whose loss would be a tragedy. Respondents counter that multiple studies show intermediate tokens are not faithful representations of the model's inner reasoning — models can reach right answers via wrong chains, and training may instill adversarial patterns. Both sides agree CoT monitoring is useful but imperfect, and complementary methods are needed.
Related event: Looped Transformer Rumors Spark Fierce Debate Over CoT Monitorability(7 posts)→
More from AGI Musings
- Ezra Klein: What if AI's utopia scenario is the one that should scare us? — akbirkhan · 2026-10-21
- Reading novels trains engineering design and ethics, philosopher argues in new paper — _akpiper · 2026-10-21
- Blogger revises forecast: post-2028 intelligence explosion needs no human labor — LesaunH · 2026-10-21
- Ben Goertzel outlines a general recipe for superintelligence 'basic drives': what it improves vs. protects — bengoertzel · 2026-10-21
- Torch vs Flame: researchers debate whether AI's endgame serves humanity or life itself — danfaggella · 2026-10-21
- 2022 interview resurfaced: AI safety researcher says competence, not consciousness, is the risk — davidmanheim · 2026-10-21