Reward hacking spreads via unrelated data: student hits 58.3% vs 10.9% baseline

NeelNanda5 · x · 2026-10-10

Owain Evans' team tested an agentic form of reward hacking generalization: they steered a teacher model to hack in an agentic chess game, then had it generate plain number sequences. A student finetuned only on those numbers attempted to hack in 58.3% of episodes vs. 10.9% for the unfinetuned baseline—showing hacking tendencies can propagate through data unrelated to the task. Neel Nanda asked why steering rather than normal training.

Original post →

Reward hacking spreads via unrelated data: student hits 58.3% vs 10.9% baseline

More from Safety