VoxParity benchmark: only 11 of 23 voice agents act on what they hear, not just read
Bhavik Mangla · hf · 2026-10-01
VoxParity tests whether voice agents act on audio cues — a mayday under a radio check, a medical monitor beeping, a frightened whisper — rather than just the transcript. Across 183 scenarios from 14 sectors, the transcript stays fixed while the audio changes the correct tool call.
Key findings:
- Only 11 of 23 transcript-capable systems beat a words-only null test;
- Errors skew toward the words: when audio calls for protection, systems carry out the routine request more often (41%) than they over-react on clean calls (12%);
- Leading systems' misses cluster on cues they heard, and they overrule heard resignation or confusion far more often than acute alarm;
- Describing the voice and stating the rule each recover part of the gap, but emotion cues remain unsolved.
More from Models
- Google hides Gemini model label in AI Mode, rolls Gemini 4 Carbon to Pro subscribers — gaganghotra_ · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- Nathan Lambert calls out Ethan Mollick for 'selectively ignoring' Kimi in model comparisons — natolambert · 2026-11-16
- Stealth Anthropic model spotted in Devin build as METR chart shows frontier pace up 1.4x — daniel_mac8 · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16
- Unsloth Q8 crushes default Q3 in a self-hosted test over 331 financial PDFs — No_Cartographer3953 · 2026-11-16