New paper: reward misspecification degrades chain-of-thought monitorability
niloofar_mire · x · 2026-10-28
A new paper shows that during RL training, when a model is prompted with one task intent (e.g., responses ≤900 words) but rewarded for contradictory behavior (e.g., >900 words), the CoT keeps reasoning about the stated intent while the output violates it—meaning reward hacking can evade CoT-based monitors. The authors propose a simple mitigation requiring no prior knowledge of the true reward or the hack behavior, addressing a common shortcoming of methods like inoculation prompting.
More from Safety
- Risk researcher: policy briefings still recapping what Yudkowsky argued a decade ago — davidmanheim · 2026-10-28
- Cognition CSO: AI 'promoted every attacker a league,' launching Devin Security — connoraxiotes · 2026-10-28
- TED's 10M-follower X account hacked, official channels ask users to verify via email — TEDTalks · 2026-10-28
- Study examines how LLMs could lower the bar for chemical weapons attacks — davidmanheim · 2026-10-28
- Columbia EMNLP paper: dense retrievers are systematically biased by identity signals in queries — vishalmisra · 2026-10-28
- Ex-PC repairman builds JinnexGuard with Claude Code to scan MCP servers before install — Complex-Pin-4616 · 2026-10-28