A frontend-backend architecture for tool calls in full-duplex speech models
Ke Hu, Slyne Deng, Chen Chen, Elena Rastorgueva, Edresson Casanova, Punit Kumar, Dharmendra Choudhary, Nikhil Srihari, Ameya Sunil Mahabaleshwarkar, Viet Anh Trinh, Slim Essid, Oluwatobi Olabiyi, Zhehuai Chen
cs.CL
2026-09-17
NVIDIA splits duplex speech from tool use: frontend emits a delegation token and streams ASR to a text LLM. Single-turn recall is 92–97%, with 81.2% rejection of irrelevant calls.
Full-duplex speech-to-speech models already listen and talk at the same time, handle barge-in, and switch turns like a person. Tool use is a different skill. On τ-Voice, leading commercial duplex voice models finish only 31%–51% of grounded customer-service tasks in clean conditions. GPT-5 as a text agent hits 85% on the same tasks. Noise and accented speech widen the gap.
Stuffing tool calls into a Moshi-style audio-native duplex model hits a capacity wall. Audio tokens spend parameters and context that a text LLM would spend on facts, instruction following, and tool protocols. Several concurrent systems keep a speech frontend and hand reasoning to a text backend: KAME, MoshiRAG, Thinking Machines' interaction-background setup, Qwen-audio-agent, GPT-Live. How the frontend signals delegation, and how backend text re-enters the duplex loop, is mostly undisclosed.
This NVIDIA paper reduces the interface to a control token on the frontend text channel, with almost no architecture change.
The frontend is a duplex speech-to-text model; speech out goes through a separate streaming TTS. User audio is encoded by a 600M-parameter Parakeet streaming encoder and fed to NVIDIA Nemotron-Nano-9B-v2-Base. One backbone decode produces two streams: a streaming ASR transcript and agent text. VoiceChat-TTS takes explicit turn-start and interruption tokens and emits codec speech incrementally.
When a query needs a tool, for example a weather lookup, the frontend emits <tcbos> on the agent channel, usually about 320 ms after the user stops. It then fills roughly one second of hold speech and fires <tceos>. The endpointed ASR transcript goes to the backend.
The backend is a LangGraph ReAct agent: an instruction-following LLM node plus a tools node, with a conditional edge that keeps calling until the model stops emitting calls. Multi-turn state lives in a thread-keyed checkpointer, so each turn only sends the current ASR. The natural-language result is written into the frontend with <pfbos> / <pfeos>; the frontend is trained to repeat that text exactly, then TTS speaks it. During the call the frontend is silenced with pad tokens. If the backend emits an invalid tool request or plain text, that text is still prefilled. A misfire on the frontend can still be talked down by the larger model.
Training is pretraining then SFT. Tool-call dialogues are generated by Nemotron 3 Nano, Gemma-4-31B-IT, Qwen3.5-397B-A17B and similar models, filtered by LLM judges, synthesized with TTS, and quality-gated by Parakeet WER/CER. Domain tracks such as airline and retail come from two Qwen3.5-235B-A22B models playing user and agent; failed trajectories are dropped. Scale is about 530k hours of pretraining, 111k hours of SFT, 16k hours of ASR data, and 8.5k hours of multi-turn tool-call speech. The system prompt randomly shows either the full tool schema or only name plus description.
Prefill spans are masked from the loss. Training also inserts about one second after <tceos> to mimic tool latency.
Single-turn evaluation uses the ServiceNow-AI spoken BFCL split, scored by AST. With Qwen3-30B-A3B and an external ASR, mean AST is 74.6% and irrelevance rejection is 81.2%. GPT-realtime sits at 80.8% and 90.8%. The widest gaps are Parallel-Multiple (61.1% vs 74.0%) and Irrelevance. A 7B backend already averages 71.7%, above Ultravox-v0.6 Llama-3.1-8B at 43.3%; that Ultravox run called the offered function on all 240 irrelevance prompts. Delegation-token recall is 97.2% / 92.0% / 95.0% / 93.5% on Simple, Multiple, Parallel, and Parallel Multiple. Switching from internal to external ASR lifts the 30B mean from 73.0% to 74.6%, and Parallel-Multiple from 55.1% to 61.1%.
| System | BFCL mean | Irrelevance |
| GPT-realtime | 80.8 | 90.8 |
| Ultravox-v0.6-32B | 76.6 | 82.5 |
| Ours-30B-extASR | 74.6 | 81.2 |
| Ours-7B-intASR | 71.7 | 76.7 |
| Ultravox-v0.6-8B | 43.3 | 0.0 |
Full-Duplex-Bench v3 is 100 real-human scenarios from 12 speakers, with pauses, repairs, and self-corrections. With Qwen3-235B-A22B: tool accuracy 71.7%, argument accuracy 55.2%, Pass@1 48.0%, response quality 67.0%, turn-taking 100%. GPT-realtime-mini is Pass@1 51.0% and Res-Q 62.0%; full GPT-realtime is 61.0% and 74.7%. Interruption rate 51% and filler rate 83.3% dwarf GPT-realtime's 13.5% and 12.1%. Fillers are intentional hold phrases. Interruptions spike because the frontend backchannels at pauses; it keeps listening, so the later tool call usually still sees the full request.
EVA-Bench covers 213 airline, ITSM, and medical-HR tickets. Scoring uses agent text, not TTS. The 235B backend reaches EVA-A 46.6 and task completion 57.3%, above GPT-realtime-mini at 33.1 / 37.1, close to Gemini 3.1 Flash Lite at 45.6 / 57.1, still below GPT-realtime2 at 59.4 / 68.1. Same-size check: the 30B backend scores EVA-A 32.4 and task 40.4%, versus Qwen3-Omni-30B-A3B-Instruct at 29.1 / 32.5. Airline completion is 72% against GPT-realtime2's 54%. ITSM is 56.3% vs 75%, medical HR 49.4% vs 69.9%. Those two domains need chains of 6–8 successful calls; one missed delegation token breaks the chain.
Ordinary duplex skill mostly holds after tool-call SFT. Internal turn-taking precision/recall is 82/91 against 85/94, latency 438 ms vs 423 ms, barge-in accuracy 99% vs 100%. OpenbookQA is 64.7 vs 65.0. Streaming ASR WER rises from 10.80% to 11.47%. On Full-Duplex-Bench v1 the model answers faster: smooth turn-taking TOR 100% vs 93%, latency 92 ms vs 221 ms. Pause handling gets worse, Pause TOR 68.2% vs 53.2%.
The split is modular. The speech frontend keeps listening, talking, and barge-in. Tool use and multi-hop planning sit on a text LLM that already knows how to call functions. Swapping 7B, 30B, or 235B backends does not require a frontend redesign. For a team shipping a voice agent, this is cheaper than teaching tool protocols inside audio tokens.
It is not an upper bound on end-to-end spoken agents. BFCL mean and irrelevance both trail GPT-realtime, and FDB3 Pass@1 does not beat GPT-realtime-mini. What is clear is the interface: one delegation token, a prefill-and-repeat loop, and a swappable backend.
Long tool chains are the weak point. ITSM and medical HR need 6–8 successful calls in a row, and completion falls well behind GPT-realtime2. One frontend miss cannot be recovered by a stronger backend.
The eval is text-heavy. EVA and FDB3 response quality use agent text, not synthesized speech run back through ASR, so TTS distortion is out of the picture. End-to-end latency is omitted because backends mix local vLLM and cloud APIs. FDB3 filler and interruption rates are too high to read as more natural; they are design choices.
Most tool-call audio is synthetic. The authors blame the WER rise and worse pause handling on that mix; the SFT set has almost no natural pauses. Misfires are supposed to be caught by the backend, but the paper reports positive recall only, not a standalone false-positive rate.