Automated Evals Recover 87% of Human-Flagged Failures, Parlance Labs Study Finds
HamelHusain · x · 2026-09-19
Parlance Labs (Hamel Husain et al.) published a study, "Do Automated Evals Work?", benchmarking automated eval agents—Braintrust's Loop, Arize's Alyx, LangSmith's Engine, plus general coding agents—against manual error analysis.
- Setup: 100 production traces from an apartment-leasing AI, hand-labeled by a domain expert, then labels masked so each system had to find failures on its own.
- Results: The best system recovered 87% of human-flagged failures, and every system surfaced issues humans had missed.
- Limitation: Fully automated approaches consistently missed interactions that "looked correct" but delivered a poor user experience.
- Advice: Use agents to help, but keep humans in the loop. When reviewing large traces, focus on the first upstream failure and make relevant evidence easy to inspect, expandable as needed.
- The authors also highlight "criteria drift": you need criteria to grade outputs, but grading is how you discover them—hence their long-standing caution against fully outsourcing evals to LLMs.
More from coding & agent
- Dev launches Jev Search: free open-source tool that picks where to search and ranks results — gaganghotra_ · 2026-09-19
- Ex-Meta Llama 3 RL lead joins Merrai, an AI memory-layer startup, as advisor — misovalko · 2026-09-19
- Braintrust adds Jev as a judge scorer: typed decisions at up to 193.6× speed and 444.6× lower cost — multiply_matrix · 2026-09-19
- Chrome team publishes a framework for designing WebMCP tools for agentic workflows — gaganghotra_ · 2026-09-19
- I spent $3.40 on Jev in 24 hours: it will be Jev + LLMs, not Jev vs LLMs — gaganghotra_ · 2026-09-19
- Agentic Benchmark Checklist paper shows flawed agent benchmarks skew results by up to 100% — ddkang · 2026-09-19