Automated Eval Systems Recall 87% of Human-Labeled Failures but Miss Nuance
HamelHusain · x · 2026-08-18
Parlance Labs released a study comparing human-annotated traces against automated eval systems from Braintrust, Arize, and LangSmith. Using 100 production traces from an apartment-leasing AI, the best automated system recovered 87% of failures flagged by humans and found issues humans missed. However, fully automated approaches consistently failed to catch interactions that "looked correct" but provided poor user experience. The conclusion: while tools like coding agents can help, fully automated evals without human involvement are not yet viable due to nuances in user experience.
More from Research
- Kimi K3 Kernel Achieves 2.05x Speedup Over Official FlashKDA — ChengleiSi · 2026-08-19
- Nature paper: Epigenetic drift in microglia drives aging — rand_longevity · 2026-08-19
- NeurIPS Competition SimulacraBench: AI Survey Prediction — soumitrashukla9 · 2026-08-19
- AI for Science: Estimating Bacteria Size with Code — johnowhitaker · 2026-08-19
- UAI 2026 paper: identifiability metrics show systematic false positives in interpretability evals — RexDouglass · 2026-08-19
- GenBio AI releases AIDO Cell preview: one world model simulating the whole human cell — AllThingsApx · 2026-08-19