Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
Ilgee Hong, Changlong Yu, Zhenghao Xu, Xin Liu, Yuwei Zhang, Qin Lu, Bing Yin, Tuo Zhao
cs.CL, cs.LG
2026-09-30
Preference rationales serve as a privileged teacher for self-distillation; entropy-shift masking then trains judges that beat Dr. GRPO by 2-9 points on subjective benchmarks.
The standard recipe for training an LLM judge is outcome-supervised RL: a GRPO-style algorithm whose only reward is whether the final verdict matches the gold label, one scalar credited to every token in the rollout. That is fine for objective tasks such as math and coding, where the decisive criterion is correctness and RL just sharpens execution. Subjective tasks are different. Whether a chat response is helpful depends on which criteria the judge invokes and how it weighs them, and a rigorous judge applying the wrong criterion still returns the wrong verdict. RL is also known to push policy entropy down over training, which narrows the pool of criteria the judge explores.
Preference datasets already carry a signal this objective ignores: the one- or two-sentence rationale annotators write when assigning the label, which names the decisive criterion.
Self-distillation, with one model in two roles. The student sees the prompt; the teacher additionally sees the annotator rationale, a privileged context the student never sees, in training or at deployment. Training minimizes an on-policy reverse KL over the full vocabulary at every position of the student's rollout, so criterion-choice tokens finally get supervision that does not depend on verdict correctness.
Not every position carries useful signal. Define the per-position entropy shift as the student's next-token entropy minus the teacher's. A large positive shift is context sharpening: the teacher concentrates probability on one feedback-aligned expression of a criterion, so the phrase "correctly includes" pushes 0.97 of probability onto Correctness. A large negative shift is context spreading: the teacher reopens the choice across several semantically equivalent alternatives, so when the student commits to Originality, "factually verifiable" spreads mass back over Correctness and Fidelity. The authors read sharpening as memorizing a specific wording and spreading as preserving semantic understanding.
The measurements back the split. Partitioning each rollout into top-30%/middle-40%/bottom-30% by shift, both tails carry 20-28x the per-position KL of the middle. Criterion-selection spans are 16.7% of tokens but 44.4% of KL mass; criterion head-noun positions are 0.4% of tokens with 11.1% of KL mass, and 94.8% of them sit in the extreme tails.
The method is one line: within each generation, rank positions by entropy shift, mask the top ρ fraction, keep the rest. ρ = 0.7 masks the 70% of positions where the feedback makes the teacher most confident; ρ = 0 recovers naive self-distillation.
Training uses a cleaned HelpSteer3-Preference, and the cleaning matters: after filtering, about 35% of upstream rows are byte-identical duplicates, and about 918 of 1,553 validation rows are identical to train rows, which would be direct leakage. Evaluation is out-of-distribution on RM-Bench and RewardBench v2, base model Qwen3-30B-A3B-Instruct:
| Method | RM-Bench Chat | RM-Bench Math | RBv2 Factuality | RBv2 Focus | RBv2 Math | Total Avg. |
| Dr. GRPO | 74.07 | 95.59 | 71.79 | 80.00 | 87.43 | 80.31 |
| Naive SD | 76.74 | 94.45 | 78.53 | 86.26 | 81.42 | 80.17 |
| SD+mask | 80.53 | 95.63 | 79.16 | 85.66 | 84.15 | 82.50 |
On subjective subcategories both SD variants lead by 2-9 points. The objective side flips: on RewardBench v2 Math, Dr. GRPO's 87.43 beats SD+mask's 84.15 by 3.3 points. The 4B results follow the same pattern (total 76.14 to 79.04), passing trained baselines in the 7-8B range such as Think-RM-8B at 62.70. The 82.50 total clears DeepSeek-R1 (80.15), RM-R1-32B (76.85) and Nemotron-49B (77.15), and sits two points below prompted Claude-Sonnet-4 (84.50).
The mask ablations are the strongest part. Sweeping ρ from 0 to 0.7 gives 80.17/80.63/80.68/82.50, so within the tested range more masking is better. At matched ρ = 0.7, every alternative selector loses: direction flip 79.23, absolute-shift 79.73, random 79.55, student-entropy 79.57, teacher-entropy 78.61. The gain comes from keeping low-shift positions specifically, not from supervising fewer positions.
As a downstream check, each trained judge picked winners in a best-of-eight tournament on Arena-Hard-v2 creative-writing prompts (GLM-4.5-Air generations, Claude Sonnet 5 scoring). SD+mask selected responses with a 0.632 win rate, 7.6 points above the Dr. GRPO judge (0.556) and 2.0 above naive SD (0.612).
The paper pins down a structural gap in RL-trained judges: credit assignment hangs entirely on the final verdict, so criterion selection gets no signal, and subjective tasks are decided at exactly that layer. The fix consumes rationale text that preference datasets already carry, so there is no new annotation cost, and the mask is a per-generation quantile operation with near-zero implementation cost. Teams training reward models can adopt this directly; code is at horizon-llm/OPSD-GenRM. In scope this is a method improvement plus a careful mechanism analysis, not a paradigm shift, and the subjective gains do not transfer to objective categories.
The authors list two: the method depends on feedback quality, and a rationale that fails to name a decisive criterion gives the mask nothing to protect; and known self-distillation issues such as hallucination and instability in long-chain-of-thought models are not addressed, with only instruct models tested.
Open questions from the reading. The sharpening-as-memorization story rests on paired embedding-similarity differences of +0.066 and +0.037, which are small, and the semantic annotation covers 158 positions across 102 rollouts. The downstream test stops at best-of-eight selection and never runs a full judge-guided GRPO loop, which the authors acknowledge. ρ is only swept to 0.7, training draws on a single dataset, and the long-rubric feedback variant lives in an appendix without numbers in the main text.