Netflix details its production LLM judge: hundreds of thousands of recommendations scored weekly
omarsar0 · x · 2026-08-24
Netflix shares how it keeps an LLM judge effective in production, scoring hundreds of thousands of show-level recommendation explanations per week for millions of mobile members.
The judge is framed as a lifecycle with four phases rather than a validate-once artifact:
- Birth: define multiple evaluation criteria and build curated benchmarks with human labels and rationales.
- Training: refine the rubric via Reasoning-Aligned Rubric Tuning, using a meta-judge over reasoning output as the learning signal.
- Deployment: one judge in two roles — quality gating and reflective generation.
- Monitoring: continuous human-in-the-loop alignment that detects drift and triggers re-tuning behind a review gate.
A five-week A/B test over tens of millions of members shifted viewing behavior.
More from coding & agent
- Developer habits shift: Agents become collaborators from simple tools — latticecut · 2026-08-24
- Dev bottleneck shifts from writing to reading code: exe.dev co-founder — thursdai_pod · 2026-08-24
- The biggest AI mistake: trying to reinvent the wheel instead of using tools — Tired40s · 2026-08-24
- DeepPaperNote turns research papers into Obsidian notes — tom_doerr · 2026-08-24
- Open Source RobotSoul: Persistent Identity for Agents After Context Resets — robauto-dot-ai · 2026-08-24
- Hermes Agent Masterclass Module 5: Providers & Local Models — NousResearch · 2026-08-24