Are LLMs Reliable Judges? 8 Months of Lessons Learned
larabyeol · reddit · 2026-07-07
A team processing roughly 50,000 LLM outputs weekly shared 8 months of practical experience using GPT-4o for automated evaluation. It proved fast and effective at catching obvious regressions, formatting errors, and hallucinated entities. However, it suffered from poor cross-run consistency (scoring divergence exceeded 30% within two weeks without locked prompts and model versions), positional bias (preferring the first option in A/B tests), and leniency toward its own outputs. They recommend locking evaluation prompts and model versions, as well as swapping positions across two runs to mitigate these issues.
More from Research
- NUS builds a soft force sensor that drives actuators without electronics or power — CurieuxExplorer · 2026-07-27
- Chelsea Finn says robot RL is bottlenecked by physical rollout cost, not algorithms — ycombinator · 2026-07-27
- ICML 2026 oral paper replication scores stay middling after a stricter re-scoring — profjamesevans · 2026-07-27
- Long-running agents will need immutable event logs, this thread argues — sebpaquet · 2026-07-27
- Seed IQ navigates Doom II, prompting questions about benchmarks beyond ARC-AGI — Fit_Transition8824 · 2026-07-27
- Agentic Data Science in Practice: Agents Write Code but Answer Wrong Questions — hugobowne · 2026-07-27