UC Berkeley's RaRE Shows Judge Ensembles Beat Single Judges in Rubrics-as-Rewards RL
YutongBAI1002 · x · 2026-10-29
UC Berkeley researchers (Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell, Yutong Bai) released Judging the Judges: Confidence and Disagreement in Rubrics as Rewards.
- Context: Rubrics as Rewards (RaR) uses an LLM judge scoring against a structured rubric to extend RL to non-verifiable, open-ended tasks; prior work focused on improving the rubric or a single judge, not on which judges to use or how to combine them.
- Finding: Judge identity alone causes substantial score spread when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges.
- Method — RaRE: judges score rollouts 1–5 Likert; top-5 logprobs yield an expected rating and per-judge confidence. (1) A soft-Likert reward weights judges by inverse-variance confidence; (2) a drop-on-disagreement mask excludes high-disagreement rollouts from the GRPO update.
- Results: naive averaging of judges fails to beat the best single judge, but RaRE outperforms every single-judge variant on both HealthBench and GPQA Diamond, inheriting complementary strengths of its constituent judges — establishing judge ensembling as a compelling axis for improving RaR.
More from Research
- Preprint: Human peer reviewers caught only 27% of major errors, all 15 AI models beat them (53-96%) — anshulkundaje · 2026-10-29
- New paper rules out Hamiltonian Phase States as a route to quantum-only Microcrypt — StefanoGogioso · 2026-10-29
- MSR AI for Science team hiring postdoc to generate molecular data for Skala — FrankNoeBerlin · 2026-10-29
- ML-enhanced resonant soft X-ray scattering enables sub-monolayer tracking of buried-interface molecular states — carbon-lab · 2026-10-29
- Any3R: one feedforward model handles 13 camera types for 3D reconstruction, beating OmniVGGT and PanoVGGT — zhenjun_zhao · 2026-10-29
- GeoGSplat: geometry priors top DTU and NRGBD with just 65K Gaussians — zhenjun_zhao · 2026-10-29