Inverse RL Helps Align AI by Imitating Humans
Michał Wiliński, Liu Leqi, Chirag Nagpal
cs.LG
2026-07-28
PARED recovers an inspectable reward from demonstrations alone, no preference labels needed, via a low-dimensional discriminator; it wins 63.4% of best-of-N reranks and 88.4% after SFT.
Aligning language models has two mainstream routes: supervised fine-tuning on demonstrations, or preference comparisons to train a reward model for RLHF or DPO. Both underuse expert demonstrations. SFT makes the model imitate the demonstrated strings, but it does not recover an explicit objective that can be inspected, reused, or optimized with policy gradients. Properties the demonstrations carry beyond verbatim imitation, like being more concise or safer, go unextracted.
The authors ask a question that has been left open: can demonstrations alone yield a reward that is visible and optimizable?
PARED (Projected Alignment Reward Estimated from Demonstrations) borrows from inverse reinforcement learning. Rather than discriminating on raw text, it maps each response to a small set of practitioner-chosen low-dimensional features, then trains a lightweight discriminator to separate expert demonstrations from the policy's own samples in that feature space. The discriminator's expert-likeness score is the log reward, r = log D.
In this instantiation the features are only seven-dimensional: two helpfulness and harmlessness scores from gemma-3-27b-it (each divided by 10), plus five LDA topic coordinates. Response length is deliberately excluded because it becomes a controllable shortcut under policy optimization. The discriminator is logistic regression over a degree-2 expansion (raw coordinates, squares, pairwise products), so it is very lightweight.
The key difference from a standard reward model is the source of supervision. A reward model needs dedicated preference annotations, while PARED needs only demonstrations, which can be augmented with AI feedback as extra dimensions. Because the reward lives on named features, it is auditable by construction: you can see which feature and which interaction is driving the score.
The recovered reward has two uses: inference-time best-of-N reranking, picking the highest-scoring candidate without touching weights; or as a KL-regularized on-policy RL reward, optimized with GRPO.
The dataset is built on Anthropic/hh-rlhf, with expert demonstrations generated by GPT-5.1 under audience-specific prompts for adult and child audiences; 4,000 training and 1,998 test. The base policy is Qwen2.5-3B-Instruct with LoRA.
On held-out data the discriminator separates GPT-5.1 demonstrations from GPT-OSS-20B policy samples with AUC 0.849. A same-model control (both sides GPT-5.1, only the demonstration prompt differs) still reaches 0.616, confirming the score tracks demonstrated behavior rather than generator identity.
| Setting | Win-rate vs base |
| SFT (4,000 demos) | 81.1% |
| Best-of-N reranking (1 of 16) | wins 63.4% |
| PARED from Instruct (online disc.) | 84.6% |
| PARED on top of SFT (frozen) | 88.4% vs its own SFT init |
| PARED (500 demos, from Instruct) | 70.7% |
Two distinctions matter. Ab-initio PARED reaches 84.6% against the base, slightly above SFT's 81.1%, but wins only 31.7% head-to-head against SFT, so it does not replace SFT. The natural recipe is to SFT first and squeeze the demonstrations, then optimize the inferred reward on-policy, which is what the post-hoc runs do, winning 88.4% against their own SFT initialization. Decomposed by audience, both adult and child conditions improve, so the gain is not carried by one audience.
For alignment practitioners, PARED offers an option between imitating demonstrations and training a reward model from scratch: with only demonstrations you recover a cheap, interpretable, reusable reward and skip the cost of collecting preference labels. It also enables contextual alignment, where one policy, shaped by a different reward per audience, produces audience-appropriate behavior. The most practical use is not to replace SFT but to stack on top of it and extract the signal SFT left behind.
The authors position it honestly as classical adversarial imitation learning applied to LM alignment, not a new paradigm. The boundaries are real: demonstrations here are frontier-model completions, not human data; the discriminator signal weakens by construction as the policy approaches the demonstration distribution; downstream quality is judged by a single order-swapped LLM judge (gemini-2.5-flash), so part of the gain may be judge-preferred style or length, and the intervals reflect only finite-evaluation noise. It is tested on one base model, one feature instantiation, and two audiences, so generalization is unverified. More fundamentally, the policy still optimizes a learned proxy reward, so reward over-optimization remains a risk.