Harvard/MIT paper: CoT controllability fails due to entropy collapse in first tokens
rohanpaul_ai · x · 2026-11-04
A Harvard and MIT paper traces why reasoning models can't steer their chain-of-thought: entropy collapse in the first few tokens after the thinking tag (e.g., habitual "Okay" openings). Two inference-time interventions recover controllability — prefilling (even a single token helps) and banning frequent opening tokens plus in-context examples, boosting average compliance by 34–41 percentage points. Addition/style controllability and suppression controllability behave as separate capabilities, with only the former consistently correlated with CoT monitorability — a caveat for CoT-based safety monitoring.
Related event: Harvard/MIT Paper: Early-Token Entropy Collapse Limits CoT Controllability(4 posts)→
More from Safety
- Tracking which AI tools can access sensitive data is an unsolved enterprise headache — Ian-Cubeless · 2026-11-04
- Dataset of 2k+ LLM chats with real voters released to build better decision evals — logangraham · 2026-11-04
- Reuters: OpenAI Staff Warned for Years About Security Lapses in Its Monorepo — rohanpaul_ai · 2026-11-04
- Interview: how AI is changing the economics of privacy — _ghnn_ · 2026-11-04
- David Krueger: time for a literature review of AI pause proposals — DavidSKrueger · 2026-11-04
- Yoshua Bengio urges safe-by-design AI research, robust regulation and multilateral cooperation — Mila_Quebec · 2026-11-04