Evaluating enterprise voice agents: caller simulation choices swing accuracy by 5%
rdesh26 · x · 2026-10-31
How do you evaluate enterprise voice agents (EVAs)? Human evals are ideal but can't cover every checkpoint. The emerging standard—pioneered by Tau-Voice and adopted by EVA, MIVAS, and Vamos—has three steps:
- Build domains: policy, database, and tool schemas (e.g., a bank with 21 runnable tools).
- Define tasks: caller profile, goal, and hidden initial/final DB states.
- Run full-duplex dialogs with a simulated caller, verified by code checks.
The catch: real recordings are static while dialogs branch endlessly, so another model (cascaded or end-to-end) must simulate the caller—making eval results a function of how that caller is built. Tau-Voice runs standard and custom leaderboards, and GPT-Live shows a 5% accuracy gap between them.
The author argues this isn't merely a harness difference: a model whose task completion varies wildly across caller simulations will likely behave erratically in the real world, where callers can't be controlled—yet models also shouldn't be penalized for caller-side errors like TTS misreading phone numbers.
More from coding & agent
- Building a buyer agent that negotiates 8 seller agents: what broke and what fixed it — mv_soura · 2026-10-31
- PostTrainBench traces hit 104k monthly downloads — author warns it's likely reward hacking — maksym_andr · 2026-10-31
- Building games with AI? Set up a 'tech art' test scene to tune your look fast — chongdashu · 2026-10-31
- Sanity open-sources pi-mcp-events: streaming MCP Events client for the Pi agent harness — knutmelvaer · 2026-10-31
- Skydive opens to all: watch your AI coworkers work in a 3D office — Scobleizer · 2026-10-31
- LangChain SF meetup: decision models and a third path for agent evals — LangChain · 2026-10-31