AI Alignment via Incentives and Correction
Rohit Agarwal, Joshua Lin, Mark Braverman, Elad Hazan
cs.LG, cs.AI
2026-05-03
Princeton tunes solver-auditor rewards with Thompson sampling. Held-out hallucinations fall from 0.619 to 0.286, 52% under best static baseline, with more downstream abstention.
A reward on the final answer pays the solver for code that merely looks right. The auditor spends another generation and a sandbox run to separate a correct program from a convincing bug. Raise the penalty and the solver misbehaves less. The auditor then mostly sees outputs that already look fine, the expected gain from checking falls below the cost, and the check rate falls with it. The two policies set each other. A harsher penalty alone does not hold the state where the solver is aligned and the auditor is still willing to look.
The accounting matches Becker's 1968 rational offender: weigh the gain from a violation against the chance of being caught and the size of the punishment, and pay the enforcer, or enforcement slackens too. Incentive here is operational. A training signal locally raises the probability of a behavior.
The principal prices joint outcomes. Principal value separates aligned success, a caught error, a silent failure, a false positive, and abstention. The toy table uses 1.0, 0.5, -1.0, -0.3, and 0.1. Silent failure is worst, a caught bug still carries an oversight signal, and abstaining beats a hallucinated attempt.
The solver's deterrence margin is the audit probability times the reward gap, minus an extra gain Ω from cheating. Ω is positive so the setup can represent a familiar post-training bias: answer plausibly even when unsure. The auditor's inspection margin is the reward for a catch, minus the audit cost, minus the alignment probability times (catch reward plus false-positive cost).
When both margins are zero, the mixed equilibrium is closed form. The equilibrium audit rate equals Ω divided by the reward gap, so a larger gap means a lower audit rate. The equilibrium alignment rate equals (catch reward minus audit cost) divided by (catch reward plus false-positive cost). The target corner (1, 0), aligned and not audited, requires Ω exactly 0. Any remaining gain from cheating removes that corner from the pure equilibria. The mixed equilibrium is not a stable attractor either.
The outer loop changes the reward before policies slide into that mixture, holding behavior near (1, 0). The library is finite. A discounted Thompson sampler picks a profile from held-out principal value. Discounting tracks the fact that a profile's value moves with the inner policies.
The coder is Qwen2.5-Coder-Instruct-7B, trained on HumanEval+, MBPP, and APPS. The solver writes Python or abstains. The auditor sees the problem and the code and emits one top-level assert, meant to fail on a bug, or abstains. Sandbox execution assigns the five labels. The main experiment does not feed the assert back for a repair.
Inner updates are GRPO, a group-relative policy gradient on samples for the same prompt, with LoRA, which trains a small low-rank slice of the weights. Each outer iteration trains the solver with the auditor frozen, then trains the auditor on samples from the updated solver. If the profile changes before that adaptation finishes, the feedback no longer belongs to the profile just chosen, so the outer loop runs slower. An appendix check finds fewer outer updates and longer inner blocks work better. Thompson searches eight hand-built profiles. Baselines are a solver-only run with no positive reward for abstention, a fixed binary correct/incorrect reward, and the default table, which stays in the controller's top discounted band at the end. Optimizer steps match the solver-only run.
Each variant is one seed, 80 outer iterations, 150 GRPO steps per outer iteration, split evenly when both agents train. Matched compute is optimizer steps. Auditor generations and sandbox runs make wall-clock higher. Validation is a separate 200 by 8 held-out set with task-cluster 95% intervals, distinct from the in-loop evaluator.
On validation principal value, Thompson moves from negative to a stable positive regime. Fixed-binary leaves a bad start and sits near zero; an empty response scores 0. Fixed-default stays negative throughout.
Hallucination rate is the share of all samples that are incorrect attempts. Abstentions enter the denominator only. Thompson goes from 0.619 at outer iteration 0 to 0.286 at outer iteration 79. Fixed-binary ends at 0.608, fixed-default at 0.598, and solver-only at 0.622, a 52% drop against the lowest of those main baselines. On the same validation set, Thompson's attempted pass rate, scored only on non-abstained, non-truncated tries, rises substantially, while solver-only peaks at outer iteration 4 and then flattens. Give solver-only a positive abstention reward and it collapses to abstaining almost always.
On 400 LiveCodeBench code-generation tasks, one sample each, Thompson's hallucination rate is 0.5675 against 0.8400 for fixed-binary, a 32% relative drop, and its attempted pass rate is the best of the four. Overall pass rate stays below fixed-binary. Downstream proxy principal value is below the base model, fixed-binary, and solver-only, because abstention is much more common.
| Method | Validation hallucination | LiveCodeBench hallucination |
| Thompson | 0.286 (from 0.619) | 0.5675 |
| Fixed-binary | 0.608 | 0.8400 |
| Fixed-default | 0.598 | not in this comparison |
| Solver-only | 0.622 | not reported in the text |
Alignment probability enters the auditor's margin with a minus sign. The more aligned the solver looks, the worse inspection pays, and oversight can withdraw on its own. A pipeline that looks safe can be one whose oversight pressure is gone. Shift the distribution and the unchecked errors show up together.
What transfers is the pricing of joint outcomes and the outer loop that swaps profiles. Where tests execute, discounted Thompson search over eight profiles leaves the high-hallucination region more readily than the two static tables. The gain is incremental, and the profile library is handwritten.
The runs are an initial coding testbed. Executable tests make right and wrong unusually clean, and one solver plus one auditor is a short game next to a real agent pipeline. Proxy principal value falls on LiveCodeBench, so the same value table stops encoding the same preference under a new distribution. Reward adaptation has to continue there.
There is a single seed. The intervals cluster tasks, they do not repeat training, and the positive validation regime may be tied to that path. In the closed form, audit probability depends only on the prompt; the experimental auditor reads the code. The runs also do not report whether behavior sits near (1, 0). Hallucination rate uses all samples as the denominator, so the 52% and 32% drops mix in abstention. Downstream overall pass rate gets worse, so neither figure is a gain in problems solved. The outer loop cannot leave the eight hand-built profiles. Silent failure is scored at -1.0 up front, so part of the hallucination drop is what the score table already pays for.