Microsoft Research turns LLM judges into coaches and beats rubric-based RL
MicrosoftResearch · hf · 2026-07-21
Core idea
Microsoft Research proposes Experiential Learning (EL) as an alternative to rubric-only RL for open-ended tasks.
Why it matters
- Standard RL with an LLM-as-a-Judge compresses rich textual feedback into a single scalar reward.
- EL reframes the judge as an LLM-as-a-Coach that turns each on-policy response into transferable experiential knowledge.
- That knowledge conditions a teacher model and is internalized by the policy via on-policy context distillation.
Findings
- Compared with scalar rewards, EL provides denser supervision and preserves fine-grained preferences among high-quality responses.
- Across two policy families, and using feedback from either the policy itself or a proprietary model, EL consistently beats rubric-based RL on held-out and unseen open-ended tasks.
- It generalizes better beyond the training distribution and reduces reward hacking.
Related event: Microsoft Proposes LLM-as-a-Coach for Non-Verifiable Tasks(4 posts)→
More from Research
- New review: how prefrontal cortex dynamics turn world models into flexible behavior — AnnaCiaunica · 2026-10-27
- LIBERO-MAX: Robot Policies Drop 11–26 Points When the World Changes Mid-Task — drmapavone · 2026-10-27
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27