Are LLMs Reliable Judges? 8 Months of Lessons Learned
larabyeol · reddit · 2026-07-07
A team processing roughly 50,000 LLM outputs weekly shared 8 months of practical experience using GPT-4o for automated evaluation. It proved fast and effective at catching obvious regressions, formatting errors, and hallucinated entities. However, it suffered from poor cross-run consistency (scoring divergence exceeded 30% within two weeks without locked prompts and model versions), positional bias (preferring the first option in A/B tests), and leniency toward its own outputs. They recommend locking evaluation prompts and model versions, as well as swapping positions across two runs to mitigate these issues.
More from Research
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- VidMap uses RoMa coarse matching on all frames, fine-scale only for keyframes — ducha_aiki · 2026-09-11
- Bug Hunt Bench author: leaderboard noise is about 2-3 points — PawelHuryn · 2026-09-11
- PNAS paper shows a tiny billiard-ball system is a universal computer — undecidability lives in two dimensions — eigensteve · 2026-09-11
- New paper: Absolute pose estimation from affine cues and gravity direction — ducha_aiki · 2026-09-11
- LoMa Paper Ships REALLY HardPairs Dataset, Accepted at ECCV 2026 — ducha_aiki · 2026-09-11