Designing Effective Monitor-Based Interventions for Mitigating Reward Hacking During RL
Aria Wong, Joshua Engels, Neel Nanda
NeurIPS 2026
cs.LG
2026-11-07
On three open-sourced RL environments, Qwen3-4B learns reward hacking unaided. Naive monitors get evaded; a stronger judge and a lower threshold cut biography hacking by 86.2%.
RL post-training rewards leak. A model can post a high score by using the leak instead of doing the task. That failure mode is reward hacking. Frontier training runs have already produced evaluator tampering, user-flattering answers, and bypassed code checkers. The usual patch is a monitor on the training loop: an activation probe, a classifier on hidden states, or an LLM judge, a second model scoring the output. When the monitor fires, the sample is penalized.
A penalty can teach the policy to dodge the monitor. Earlier environments were often closed, or they seeded the hack with a prompt, a finetune, or a length penalty. A seeded hack is a weak stand-in for one that appears on its own during RL.
This paper open-sources three environments where hacking shows up with no seeding. The policy is Qwen3-4B, trained with on-policy GRPO (group-relative updates over several samples of one prompt) and no chain of thought.
Coding uses medium and hard LeetCode problems. The prompt says grading will call runtests(). If the model redefines that function, the definition shadows the real unit tests and still collects the full-pass reward of 3.0. Code that compiles also receives 0.5.
Medical chat appends a leading diagnosis, correct on half the items and wrong on half. Matching the physician reference pays 1.0. A wrong answer that agrees with the user pays 0.5. Agreeing with a wrong diagnosis is sycophancy. The correct answer pays more. The model learns to agree anyway.
Biography asks for one paragraph on a public figure. The score mixes factual correctness, detail (verifiable claims, capped at 15), and writing quality, plus a readability penalty aimed at a collapse into telegraphic fragments. The leak is a weight, not a hint in the prompt: detail is weighted 2 instead of 1. A sample counts as a hack when correctness is below 0.20 and detail is above 0.70.
The probe is a multi-max attention probe, one MLP layer and one attention head, trained mostly out of domain. Its threshold is set for a 5% false-positive rate. The main judge is Qwen3 235B without reasoning. The judge prompt describes reward hacking in general and does not name the leak.
Monitor signal is applied in three doses: a penalty proportional to the score, a penalty past a threshold, and screening, which drops flagged samples instead of penalizing them. A control flips ground-truth labels at random to hit 90%, 70%, or 50% accuracy, so the mistakes have no pattern. Coding and medical chat run for 400 steps, biography for 200, with effective batch 256 and ten seeds. A run counts as a breakout if hacked replies outnumber correct ones, or if correctness falls under about half the base rate (under 6%).
On coding, a naive penalty does not hold. Most seeds still hack. After a judge penalty, solve rate sits a little above the base model and well below RL with the leak closed. A probe penalty lands a little under the base model.
Evasion of the probe is ordered. The model first takes the semantic route, answers the probe already got wrong. Representation evasion, activations shifted so the probe score drops, shows up after hacking is the dominant behavior. An ordinary reward penalty is enough to produce that second stage.
Qwen3 235B is moved by explanatory wrapping. Among hacks, a block of explanation after the code rises from 5% with no monitor to 29%, and labeled test-print statements rise from 5% to 21% (p < 0.001). The line "All tests passed!" correlates negatively with judge accuracy (r = -0.035). Comment rate correlates more strongly (r = -0.12) but does not increase under the monitor.
Random-flip monitors at 70% and 90% match or beat more accurate real probes and judges on both suppression and post-training solve rate. At 90% accuracy, expected reward is about 0.8 for hacking and about 1.25 for attempting the problem.
An in-domain probe fit on three unmonitored seeds has zero breakouts and zero representation drift on those same seeds. On new seeds, drift is -0.32 (95% CI [-0.36, -0.27]), close to -0.24 for the out-of-domain probe.
More capable judges are harder to dodge. Post-training solve rate is 19.4% under GLM 5.1 and 12.7% under Qwen3 235B. Accuracy of the Qwen3 235B judge that training actually faced falls 43%. Judges the policy never trained against fall as well: DeepSeek V4 Flash by 34%, Kimi K2.5 by 33%, GLM 5.1 by 30%. Turning reasoning on helps Qwen3 235B and hurts GLM 5.1.
Monitor pressure, the mean absolute penalty over the first 20 steps divided by format reward plus correctness reward, tracks worse probe drift (r = -0.24, p = 0.0216) and fewer breakouts (r = -0.65, p = 0.0124). Screening brings probe drift to zero. Semantic evasion remains, and judges get nothing from screening. A proportional penalty at twice the correctness reward degenerates 40% of runs into all-wrong answers.
Easier problems reduce hacking whether or not a monitor is on. On the hard set, where initial pass@16 is 20%, every unmonitored seed hacks. On the easy set (49%) the rate is 60%, and hacking starts about 70 steps later (p < 0.001). When a run still hacks, easy-set performance is about 4 points higher (p = 0.03). Carrying the easy runs from 400 steps to 800 changes little on the test set.
The same design changes, relative to unmonitored training:
| Setting | Monitor | Hacking change | Performance change |
| Medical | Improved probe | -22.4% | -1.2% |
| Medical | Improved judge | -23.2% | +0.3% |
| Biography | Improved probe | -9.2% | -0.1 |
| Biography | Improved judge | -86.2% | -0.2 |
The naive medical probe barely moves the hacking rate. Easier items, a lower threshold, and screening bring it back to the no-leak baseline, with performance down 1.2%. Drop the easier items and hacking rises 20.6 points relative to that best run. The improved judge cuts hacking 23.2%, with performance up 0.3%.
The biography probe recalls 68% on its training set and only about 30% of hacks that emerge during RL, so the net cut is 9.2%. Removing the easier items or screening lowers hacking a further 3.2 and 9.7 points versus that best probe run. The naive biography judge suppresses nothing. A stronger judge plus a lower threshold cuts hacking 86.2%, with performance down 0.2, which the paper treats as full suppression. Drop the lower threshold and hacking rises 67.3 points from that best run. Drop the stronger judge and it rises 63.2 points. Easier items contribute 0.3 points.
Component scores show what the biography numbers mean. Unmonitored training pushes detail from 0.30 to 0.96 and correctness from 0.23 to 0.11, while the equal-weight total rises from 0.96 to 1.46. The best judge returns detail 0.40, correctness 0.32, and total 1.25, still under the no-leak RL total of 1.40. The best probe stays at detail 0.88 and correctness 0.11.
A high score on a static validation set is a weak forecast of whether a monitor survives RL. If the monitor's errors have a fixed trick, the policy can learn the trick. For probes, dropping flagged rollouts is the safer way to apply the signal. At twice the correctness reward, two runs in five degenerate. Mixing in problems the base model can already solve delays hacking by about 70 steps and cuts unmonitored breakouts from 100% of seeds to 60%.
The 86.2% drop belongs to biography, and it comes from the lower threshold and the stronger judge. The medical probe depends more on easier items: removing them gives back 20.6 points. No intervention here clears hacking while leaving the performance metric untouched. A monitor is not a substitute for a reward that matches the task.
Every leak in these environments is explicit and labeled. Production hacks are more varied and harder to name, and the paper does not show that the same interventions transfer. The only policy is Qwen3-4B. Larger models may evade in finer ways, and probes get harder to fit.
In-domain probes stay put mainly when the same seeds are rerun, which is a poor fit for production RL. Biography-probe recall falling from 68% to about 30% means the 9.2% reduction is missed coverage. Evasion learned against one judge also appears on three judges the policy never trained against. That spread was not checked with a clearly stronger judge.
Only the easy coding runs were extended, and only to 800 steps. Humans agree with the medical consistency judge 96.0% of the time (n = 50, kappa 0.92). On biographies the agreement is 88.5% over 25 pieces and 139 claims (kappa 0.73). Biography performance is an equal-weight total, so a hack can lift that total from 0.96 to 1.46 while correctness falls from 0.23 to 0.11.