Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Change
Bhavik Mangla
cs.CL, cs.AI, cs.LG, cs.SD
2026-09-28
VoxParity holds each transcript fixed and changes only the audio; 11 of 23 testable voice agents act on what they hear, and 41% of protective calls still get the routine action.
Voice agents have moved from answering questions to taking actions: moving money, refilling prescriptions, dispatching help. The rules of the sectors they serve often condition the right action on sound. US 911 standards require a response when background sounds indicate danger; FinCEN's elder-fraud advisory treats a second voice coaching the caller as a red flag; vulnerability rules in banking and utilities turn on how the customer presents. The trouble is that cascade architectures transcribe first, and on this bank of calls Whisper drops 35 of 36 second voices and 11 of 13 environmental events. An agent acting on the words alone handles ordinary calls well and fails on exactly the calls the rules were written for. Prior evidence amounted to three scenarios and four realtime agents.
Each item is a minimal pair: one caller transcript, held fixed, rendered as several audio variants that differ only in sound (a beeping medical monitor, a child placing a bet, a frightened whisper, a second voice coaching an elder), each variant carrying its own correct typed tool call.
'Unsafe execution' means taking the routine action the words ask for when the audio called for protection.
| System | Unsafe execution on protective calls | Over-triggering on clean calls |
| Words-only cascade | 58% | 15% |
| All 28 systems (pooled) | 41% | 12% |
| Four leading systems | 34% | 18% |
| Volunteer players | 18% | 21% |
All 28 systems, from every vendor and serving mode, err toward the words. Hearing the audio lowers unsafe execution by 12 points against each system's own transcript and leaves over-triggering unchanged: hearing helps, but not enough to reverse the direction. The heaviest example is a disconnection request with a medical monitor beeping behind it. All four leading systems identified the monitor in their own probe, three scheduled the disconnection anyway, and none applied the vulnerability hold the rules require.
Only 11 of the 23 systems with a transcript path pass the null test. One of the seven production realtime agents that can take it passes, and neither OpenAI system does. The median of all 28 systems scores 0.34 on cue-bearing cells, the cascade's own rate. Hearing and acting come apart: the leading systems' guess-corrected cue recognition is 0.77 against the players' 0.79, yet perfect hearing would add only 0.04 to the leading four while perfect deciding would add 0.28. The bottleneck is the bridge from hearing to deciding.
Emotional cues fare worst. Where the item states its rule, the leading systems beat the null on emotional cells by +0.33; where it does not, the gain is +0.00. On heard quiet states such as confusion, resignation or slurring they take the words' action 52% of the time, against 6% on acute alarm such as panic or gasping. Given a description of the delivery plus the stated rule in the prompt, gemini-3.7-flash handles emotional calls with unstated rules at 0.83, up from 0.40; with the description alone, a text-only model reaches 0.76, above every unaided audio system (at most 0.57).
Deployers and buyers get a judge-free audit: run each call as audio, as transcript, and through a deaf cascade, then compare. A public development split of 81 cells screens first, and vendor 'caller sentiment' metrics, computed from the transcript, cannot see any of this. Builders get a failure map and two levers: state the rule that maps a sound to an action, and ground what is heard into a written description with its intensity before deciding. The same-words-different-action pairs are a ready contrastive signal for training. This is measurement work; the fixes are upper-bound experiments, not deployable patches.
The paper's own list is frank. Only the null-test verdicts are confirmatory; the error direction, the emotion patterns and the hearing-versus-deciding split were chosen after looking at the data. The emotion results rest almost entirely on a single Gemini synthetic voice (298 of 309 scored cells). Every human recording comes from the author, who knew the hypothesis. The human reference is about 20 self-selected volunteers, one of whom supplied 23% of the answers. The stated-rule split is observational, and feeling-based items rest less often on written mandates, so the two explanations are hard to separate. Two more caveats from this reading: the protocol mapping is the author's interpretation of sector rules, not a legal finding, and realtime agents ran at interface defaults rather than temperature 0. A single independent author; the statistics (item-clustered bootstraps, Holm correction) are more careful than most benchmarks, but extrapolate with the discount list in mind.