Dev finds his LLM judge scores worse than the agent it grades, shares regression-testing fix
jgarg27 · reddit · 2026-10-27
A developer building a search-query-writing agent with an LLM judge found the judge was less accurate than the agent it graded: the agent got 8/9 on scored cases, the judge 7/9. The cause: each judge failure was patched with a new rule never cross-checked against others, producing false fails (a correct query legitimately returning empty was penalized) and false passes (the agent answering a broader question undetected). His fixes: a fixed labeled set of past runs — including crucial "should return nothing" labels — and decoupling generation from judging by storing agent runs and re-running only the judge, making regression passes cost a fraction of a single agent run.
More from coding & agent
- Anthropic: Single agents with skills often beat subagent designs in enterprise deployments — addyosmani · 2026-10-27
- Claude Code's idle compaction annoys users — here's the setting to turn it off — JasonBotterill · 2026-10-27
- Creator builds CCTV-style video intro with Claude Code, still stuck on AI dubbing — xiaohu · 2026-10-27
- Open-Source Agent Project GEAR Trends on Hugging Face — WizardTY · 2026-10-27
- Skill∞: Recursive Self-Improvement Where Agents Optimize Their Own Optimization Skills — NationalUniversityofSingapore · 2026-10-27
- Agent identity by name is untrustworthy: 5 runtime principles for multi-agent credential safety — karc16 · 2026-10-27