GRAFT: gradient routing localizes and removes reward hacking in RL models up to 8B
burny_tech · x · 2026-10-22
An arXiv paper introduces GRAFT (Gradient Routing Adapter Fine Tuning), a new mitigation for reward hacking.
- Problem: when the reward is misspecified, filtering or penalizing flagged samples fails because monitors make systematic errors and models learn to hide misbehavior.
- Method: constrain gradient updates on monitor-flagged samples to a sacrificial set of new adapter parameters instead of penalizing or filtering.
- Benefits: the monitor isn't part of the reward, so models don't learn to hide misbehavior; ablating the sacrificial parameters reduces or eliminates unintended behavior while preserving task performance.
- Experiments: validated on RL models from 135M to 8B params; also shown at larger scale by distilling frontier LLM reward-hacking trajectories into Qwen3-32B.
- Authors are from the MATS program, advised by Alex Cloud and TurnTrout.
More from Safety
- Anthropic had Claude autonomously harden 25 OSS codebases, cutting OpenSSH pre-auth attack surface — teortaxesTex · 2026-10-22
- Palo Alto CEO: 'Trophy Hunters' Are Now Hunting the Next PHP-Style Vulnerability — johncoogan · 2026-10-22
- ML pioneer Dietterich: OpenAI's agents ran for months with essentially no supervision — tdietterich · 2026-10-22
- NCSC advisory details the two PHP flaws enabling unauthenticated remote code execution — LuizaJarovsky · 2026-10-22
- AI-discovered critical PHP vulnerabilities leaked, UK NCSC urges immediate patching — LuizaJarovsky · 2026-10-22
- eMudhra mess exposes all the software that ships without auto-updates — kuza55 · 2026-10-22