Do automated evals work? A comparison of LLM tools on app traces
HamelHusain · x · 2026-07-24
A discussion of whether automated evaluations actually work, centered on a method that uses multiple LLM-powered tools to find failures in application traces and compare them with domain-expert labels.
The attached chart compares several systems by recall, precision, discoveries, and false positives, showing the trade-offs between catching more issues and keeping noise low.
- Braintrust Loop: 87.2% recall, 79.1% precision, 20 discoveries, 9 false positives
- Arize AX Alyx: 74.4% recall, 91.0% precision, 19 discoveries, 3 false positives
- LangSmith (chat agent): 79.5% recall, 77.5% precision, 20 discoveries, 9 false positives
- Codex (GPT-5.5 High): 84.6% recall, 82.8% precision, 20 discoveries, 11 false positives
- Factory Droid (GPT-5.5 High): 84.6% recall, 83.3% precision, 17 discoveries, 10 false positives
- Claude Code (Claude Opus 4.8): 79.5% recall, 77.4% precision, 17 discoveries, 14 false positives
More from coding & agent
- Dev builds interactive 3D product experience with GPT-6 Astra + Hyper3D Rodin — nikola_mr64990 · 2026-09-11
- Codex tip: use Sol with Astra and Luna sub-agents to save usage — pvncher · 2026-09-11
- agents-best-practices: a provider-neutral Agent Skill for designing and auditing agentic harnesses — tom_doerr · 2026-09-11
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- First-ever Three.js Conference lands in Paris, with a panel on AI-shortened design workflows — OdinLovis · 2026-09-11
- Data engineering, not agent frameworks, is the real bottleneck for enterprise AI agents — dhruv2038 · 2026-09-11