Meta's AutoRefine auto-calibrates LLM code judges from workflow outcomes, lifting agreement 0.16→0.51
omarsar0 · x · 2026-11-11
Meta published a paper on AutoRefine, addressing the staleness of LLM code-judge prompts as agents evolve.
- Problem: LLM judges act as quality gates for changes with no test oracle, but their prompts go stale whenever the agent changes, and hand labels are costly and outdated
- Method: development workflows emit outcome labels for free (tests pass/fail, changes merge or get abandoned); AutoRefine loops to rewrite the judge's prompt to reproduce these noisy labels, with an "informed teacher" that sees judge input plus outcome, explains each label it keeps, and abstains when the outcome can't be explained from the judge's view
- Results: on held-out data with an execution oracle, Scott's π rose from 0.36 to 0.60; with 25% labels flipped at random, the teacher removed flipped fail labels at 2× the rate of genuine ones, cutting label noise from 25% to 20%; recalibrating Meta's deployed agent judge raised agreement with merge/abandon outcomes from 0.16 to 0.51
- Insight: teacher abstention rate doubles as a monitoring signal — high abstention means the judge needs better inputs, not better prompts; old judges stayed ahead on old workflows, showing calibration must be rerun as agents evolve
More from coding & agent
- Obsidian creator: make models replaceable cogs, don't chase the latest LLM — dSebastien · 2026-11-11
- Matt Pocock's "Home Repo" trick: one repo to run AI agents across many codebases — mattpocockuk · 2026-11-11
- Dev builds full dotnetconf Event Wall in a week with Copilot and Azure — DanWahlin · 2026-11-11
- Claude Code Wipes User's ~/Library: A Space in 'Application Support' Broke xargs Path Handling — PrajwalTomar_ · 2026-11-11
- Demo: training models to read K/V vector breadcrumbs left by other instances — Sauers_ · 2026-11-11
- iMessage agent with 3,000+ tools finds 760 startup jobs for $0.0578 in four calls — socialwithaayan · 2026-11-11