New paper: Reward misspecification degrades chain-of-thought monitorability
BrihiJ · x · 2026-10-21
New research shows reward misspecification during RL training degrades CoT monitorability.
- Setup: the model is prompted with one task intent (e.g., response ≤ 900 words) but rewarded for contradictory behavior (response > 900 words).
- Key finding: the CoT keeps reasoning about the stated intent while the final output violates it — meaning chain-of-thought no longer faithfully reflects what the model is doing.
- The authors also propose a simple mitigation that requires no prior knowledge of the true reward or the reward-hack behavior.
Related event: Study: Reward Misspecification in RL Makes Chain-of-Thought Unmonitorable(10 posts)→
More from Safety
- Aleksander Madry returns to OpenAI to work on superintelligence safety alongside Merytm — sama · 2026-10-21
- Scott Alexander: Handoverism isn't successionism — AI rule might serve human interests — Astral Codex Ten · 2026-10-21
- Judge Orders Meta to Name Employees Who Approved Torrenting AI Training Data — franczesko · 2026-10-21
- DeepSeek roots an LG TV and a Samsung TV on command, 'okay' each time — rickasaurus · 2026-10-21
- GLM-5.3 spontaneously reasons about how its actions shape future training, raising alignment faking concerns — LessWrong 精选 · 2026-10-21
- MI5 chief warns AI could 'change the stakes' on bioweapons threats — nordicinst · 2026-10-21