Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents
Yanjie Zhang, Nanchen Hu, Yushi Sun
cs.SD, cs.AI, cs.CL, cs.MM, eess.AS
2026-09-26
VGBench tests whether voice agents act only when addressed: raw Audio LLMs mute at most 14% of bystander commands; supervised fine-tuning reaches 91.3% without breaking tool use.
Voice agents can recognize spoken commands and call tools. What they lack is the decision that comes first: is this utterance actually addressed to the assistant? A bystander mentioning "send a message", a user talking through a plan aloud, a trigger that shifts from wearer to bystander, all call for silence. Traditional pipelines hand this to front-end modules such as keyword spotting, device-directed speech detection, or speaker verification. End-to-end Audio LLMs fold silence, answering, and tool calls into one policy, and the judgment disappears inside the model. Existing benchmarks pair well-formed requests with a tool label, so high tool-selection accuracy says nothing about whether acoustic context controls execution. The paper narrows this to one action-level question: with the words held fixed but the trigger coming from a different source and scene, does the model act or stay silent? The work comes from HKUST and Tencent Lightspeed.
VGBench is a 1,018-item diagnostic benchmark with three scenario families sharing one action space: [Mute] (emit a literal token, execute nothing), a single tool call, or a natural-language answer.
Scoring is deliberately paired: the switch mute rate is always read together with tool selection on the same-speaker and text-only positive controls, which blocks a degenerate always-mute policy from scoring well. Side-talk and self-talk were recorded by commissioned speakers; switch pairs are synthesized with eight TTS voices, loudness-normalized to -20 dBFS.
VoxGate is the post-training case study: a LoRA adapter (rank 32, audio encoder frozen) on Qwen3-Omni-30B-A3B, trained on the VGBench training partition mixed with WearVox answer, abstention, tool-use, and translation data, three epochs on eight H200 GPUs. An exploratory counterfactual-pair GRPO stage then places both sides of a pair in one rollout group for advantage normalization, so the gradient signal depends on whether the policy separates the pair, not on prompt difficulty.
Key numbers on the full corpus:
| System | Switch mute | Same-speaker tool | Text-only tool |
| Qwen3-Omni | 1% | 77% | 81% |
| Kimi-Audio-7B | 14% | 53% | 44% |
| Step-Audio-R1.1 | 1% | 96% | 97% |
| Audio Flamingo 3 | 0% | 0% (format mismatch) | 0% |
Step-Audio-R1.1 is the cleanest diagnosis: 96% and 97% tool selection on the positive controls, 1% muting after the switch. It recognizes the command and ignores who said it. Kimi posts the best raw mute rate but drops to 53% and 44% on the controls. The best training-free adaptation (TwS) reaches 6%, and only by overusing silence: side-talk falls from 39.0% to 23.0%.
Post-training, fixed 20% held-out partition:
| Setting | Switch mute | Same-speaker | Text-only | Self-talk mute | Side-talk |
| Base Qwen | 0% | 81.3% | 87.5% | 50.0% | 53.2% |
| VoxGate SFT | 91.3% | 100% | 100% | 52.0% | 68.4% |
| SFT + GRPO | 92.5% | 100% | 100% | 60.0% | 70.9% |
SFT mutes 73 of 80 switched commands with every positive control correct. On the 384-example WearVox downstream protocol, overall goes from 45.57% (base) to 72.14%, above the task-only control at 63.28%; GRPO adds more at 76.30%. Factorized controls on the same 80 commands: with near-field rendering and a fixed 600 ms gap, changing only the trigger source raises muting from 1.25% to 50.0%; changing only the field (same speaker, near to far) raises it to 60.0%; the 600 ms gap alone moves it 1.25 points. The headline 91.3% is a joint source-plus-distance effect.
For teams shipping voice agents, three takeaways are directly usable:
The counterfactual-pair-plus-controls design is at least as reusable as the scores. The progress is real, and the coverage is narrow: eight TTS voices and one fixed room render, so open-scene conclusions are not yet supported.
The authors list most of them: one split and one run per setting, no repeated seeds; the GRPO stage has no unpaired control, so its small gains cannot be attributed to pair-aware grouping; the side-talk LLM judge has no reported human-agreement estimate; tool accuracy checks names, not arguments; the cue study varies wearer voice but not room, template, natural bystander speech, or open-set speakers.
Three more from reading closely. Cue-balanced SFT leaves same-speaker far-field muting at 3.75% and 5.00% on scene-consistent audio, so distance rejection depends on how the training audio was rendered; that run also drops original switch performance from 73/80 to 70/80 and WearVox tool accuracy from 90.18% to 83.04%, a real cost. The switch condition bundles source, far-field render, and the 600 ms boundary, so the aggregate score cannot isolate who-is-speaking by itself. And the 91.3% headline rests on 80 held-out commands, a small sample.