Solid evals, still blind: how Conviva caught agent failures users quietly gave up on
Automatic-Mirror7324 · reddit · 2026-09-10
Conviva's analytics agent served 20 major streaming providers during the 2026 World Cup with tens of millions of concurrent users. Their eval suite scored well but caught almost nothing — because test cases only came from loud failures, while the expensive ones were silent: a conversation scoring 0.94+ on helpfulness while the agent quietly omitted an optional search parameter and the user left to do it manually.
What worked instead:
- Score what the user does next, not what the agent says: restated requests, repeated asks, scope narrowing, manual verification afterward, corrections after resolution. No human labeling, not gameable by better-sounding replies.
- Ask "under what conditions does it fail" instead of "which trace failed": slice by request type, tool parameter, user tier; statistics first, transcripts as verification. Reading traces one at a time and sampling are exactly where rare, expensive failures hide.
More from coding & agent
- Dev builds AI landscaping pipeline on DJI drone .srt telemetry files, still self-improving — doodlestein · 2026-09-10
- Two iPhones side by side: the ideal console for managing AI agents — hudzah · 2026-09-10
- Astra agents do everything by writing Python, with recursive subagents and a persistent REPL — willcb · 2026-09-10
- A Multiplayer AI Manifesto: cloud agents everyone on the team can join — sergeykarayev · 2026-09-10
- AutoResearchExam: a 24-hour benchmark finds AI research agents overfit, with Fable 5.1 edging Astra — AlexGDimakis · 2026-09-10
- HarnessRouter open-sources a self-hosted Docker container to run Codex, Claude Code and more via one API — tom_doerr · 2026-09-10