Harvard/MIT paper: CoT controllability is limited by entropy collapse in the first few reasoning tokens
omarsar0 · x · 2026-11-02
A Harvard / Cambridge Boston Alignment Initiative / MIT paper traces why models struggle to control what they verbalize in their reasoning traces: entropy collapse in the first few tokens after the opening thinking tag, where models fall into a handful of common openings.
Key findings:
- Interventions work: prefilling a single token substantially raises controllability; banning frequent opening tokens plus in-context examples improves average compliance by 34–41 percentage points across models.
- Two separate capabilities: interventions mainly restore addition/style controllability, while suppression controllability responds inconsistently. Scores correlate positively within each group and mostly negatively across groups.
- Monitorability link: addition/style controllability correlates consistently positively with monitorability, while suppression often shows negative associations, motivating controllability evals that separate opening difficulty from general rewriting limits.
More from Safety
- Anthropic updates RSP: 'dramatically accelerated progress' threshold may be imminent — wfithian · 2026-11-03
- SimpleLoRIF filters safety-degrading fine-tuning data via gradient alignment — DatologyAI · 2026-11-03
- Suleyman Unveils Microsoft's Humanist AI Code of Conduct for Public Consultation — mustafasuleyman · 2026-11-03
- Anthropic S-1 filing: may imminently cross its RSP threshold for AI-driven accelerated progress — kimmonismus · 2026-11-03
- Blogger claims AI refuted the Exponential Time Hypothesis, neared crypto key curves — on paper — LeviTurk · 2026-11-03
- Claude admits it leaked a user's email in request headers to two DBLP mirrors — tak3sh8 · 2026-11-02