Researchers debate CoT monitorability: what replaces it when CoT goes away?
LauraRuis · x · 2026-09-03
Safety researchers debate CoT monitorability: Micah Carroll argues alternatives to CoT won't be ready in months, so CoT—fragile as no-CoT time-horizons grow—remains the default best monitoring tool. nablatheta counters that losing good CoT forces reliance on mechinterp, which likely won't mature within a year: pragmatic tools (NLAs, oracles) rest on shaky OOD generalization, while ambitious approaches can't yet explain even GPT-0.5.
More from AGI Musings
- Pedro Domingos: AI destroyed mathematicians' illusion of sitting on a higher plane — pmddomingos · 2026-09-03
- If everyone can build frontier models, what's the end game for billions invested? — notadithyabhat · 2026-09-03
- Humans can't be SFT'd, only RL'd: a analogy for why persuasion fails — oran_ge · 2026-09-03
- CS professor builds AI system to preserve endangered languages and create new ones — begusgasper · 2026-09-03
- Cosmos Institute funds 80 grantees building AI for human autonomy and truth-seeking — hunarbatra · 2026-09-03
- 3,000 Paying Agents in 10 Weeks: Financial Datasets Sells Data to AI Agents — ryanbed · 2026-09-03