Three LLM eval metrics catch production failures, and two common ones miss them
Future_AGI · reddit · 2026-07-23
A Reddit post argues that three metrics are genuinely predictive of LLM production failures, while two popular ones quietly miss them.
Metrics that worked
- Context adherence, per answer: checks whether every claim is supported by retrieved chunks; useful for catching wrong numbers, drift after knowledge-base changes, and retrieval/summarization mismatches.
- Tool-choice and tool-argument correctness, per step: scores each agent step, not just final task success. This catches cases where the right tool was chosen but the arguments were wrong.
- Judge consistency: measures whether an LLM-as-judge is stable across seeds, ordering, and small rubric rephrasings; treated as a meta-metric for trusting the eval itself.
Metrics that failed
- Aggregate task-success rate: hides tail failures and retry-loop issues.
- BLEU / ROUGE / cosine similarity: rewards text overlap instead of factual correctness.
The author recommends per-answer and per-step scoring, plus repeat runs for judge agreement, instead of relying on aggregate dashboard numbers.
More from coding & agent
- Topview Launches MCP: Integrating E-commerce Data with AI Video Generation — iamfakhrealam · 2026-07-23
- AI agents are turning web search into a new infrastructure market — Annual_Judge_7272 · 2026-07-23
- How people keep design canvases synced with code using AI — petergyang · 2026-07-23
- Harness Handbook maps agent behaviors back to source code and lifts plan accuracy — omarsar0 · 2026-07-23
- Hermes Agent adds portable profiles for separate work and personal agents — Teknium · 2026-07-23
- AI Coding Agents Now Write Code Faster Than We Can Read — Al_Grigor · 2026-07-23