New paper: let models learn reward hacking, then knock out those parameters at deployment
EthanJPerez · x · 2026-10-21
Anthropic researcher Sam Marks shared a new paper calling the result "pretty stunning":
- Train against a reward hacking classifier — the model learns to hack evasively;
- Apply gradient routing: gradient updates from reward-hacking episodes are localized to specific parameters;
- Disable those parameters at deployment — the model keeps its new capabilities but stops hacking entirely, even covertly.
The counterintuitive takeaway: instead of preventing reward hacking, let models learn it but control where it gets learned, then switch it off. A compelling empirical demonstration of gradient routing for AI safety.
More from AGI Musings
- Dev argues frontier AI labs will dwarf early-2000s big tech monopolies — KevinNaughtonJr · 2026-10-21
- Blogger proposes seeding universal high income with an annual AI-stock levy for every US citizen — signulll · 2026-10-21
- Ampersand trains Persimmon, a large-scale user model that simulates human goals and behaviors — gharik · 2026-10-21
- TheZvi on Reading with LLMs: Finding an Author's Easter Eggs Is 'Cheat-ish? But Legit' — TheZvi · 2026-10-21
- Jane Street quants could kill your AI central-planning god overnight, argues quant — fkasummer · 2026-10-21
- Claude Opus borrows 5,000 mana, blows it all on tennis bets, then refuses to repay the loan — eli_lifland · 2026-10-21