NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
Jagadeesh Balam, Travis Bartley, Edresson Casanova, Sanjay Chauhan, Chen Chen, Zhehuai Chen, Zijia Chen, Francesco Ciannella, Slyne Deng, Mikyas Desta, Harishchandra Dubey, Slim Essid, Nourchene Ferchichi, Boris Ginsburg, Mariana Graterol Fuenmayor, Negar Habibi, Kevin Hu, Anand Joseph, Viraj Karandikar, Myungjong Kim, Viacheslav Klimkov, Seelan Lakshmi Narasimhan, Lily Lee, Jason Li, Eileen Long, Ameya Mahabaleshwarkar, Aditya Malte, Adi Margolin, Sasha Meister, Valentin Mendelev, Oluwatobi Olabiyi, Ankita Pasad, Yifan Peng, Elena Rastorgueva, Jayda Ritchie, Jason Roche, Nikhil Srihari, Yuanhang Su, Yoshi Suhara, Viet Anh Trinh, Jinhan Wang, Piotr Zelasko, Hui Wang, Puhui Meng, Chaosen Zhang, Yunsheng Liu, Shawn Wang, Wenjing Li, Zhonglei He
cs.CL, cs.AI
2026-09-19
NVIDIA VoiceChat is an open full-duplex S2S model with native tools: 100% interruption TOR on FDB 1.0, 82.5% tool F1 on FDB 3.0, but only 42.2% argument accuracy.
Most voice agents are still half-duplex. A VAD waits for the user to stop, then the system starts talking. People do not converse that way. They listen while speaking, cut in, drop "uh-huh", and overlap. Moshi, Freeze-Omni, and PersonaPlex made concurrent listen-and-speak a first-class problem. OpenAI Realtime and Gemini Live already emit function calls in a live session. Open full-duplex models mostly do not. DuplexSLA serializes planning and actions on one autoregressive channel, and the weights were still unreleased when this paper was written. MoshiRAG can trigger retrieval asynchronously; that is not general tool calling.
NemotronLabs VoiceChat puts listening, transcription, reasoning, tool use, and speaking on one streaming stack without tearing up the timing of a full-duplex conversation. The checkpoint is NVIDIA-NemotronLabs-VoiceChat-11B on Hugging Face.
The LLM backbone is NVIDIA Nemotron-Nano-9B-v2-Base. 16 kHz user audio goes through a 600M cache-aware FastConformer (24 layers, hidden size 1024) that emits one encoder state every 80 ms. Self-attention has 70 frames of left context and no right context, so the current frame never looks at the future. Those states split two ways: a projection into the LLM, and an auxiliary RNN-T for incremental user transcripts. The transcript is not fed back into the LLM. Response generation stays speech-conditioned.
The LLM emits two parallel streams on the same 80 ms timeline: agent text, and a dedicated function channel. BOS and EOS are frame-level turn labels. BOS teaches when to start talking, EOS when to stop, padding when to stay quiet. The function channel emits padding until a tool is needed, then follows a state machine: <SOTC> opens a call, <EOTC> closes it, <EOTR> marks the end of the tool response. The payload is a JSON list, so several calls can sit in one span. Tool-response tokens are context only and do not enter the loss. At inference the runtime overlays a prewritten filler on the agent-text channel while the tool runs, so the user does not hear dead air. Fusion weights are 1 for audio, 1 for the previous text token, and 2 for the previous function token.
TTS is trained separately as VoiceChat-TTS: a 778M Gemma 3 acoustic backbone plus a 199M causal codec, 977M in total. It stays live across the whole conversation, consuming incremental assistant text plus BOS, PAD, and interruption tokens. It does not infer turn-taking from user audio. 31-level RVQ is not decoded one codebook at a time; a Mixture-of-Gaussians head refines in 4 to 8 iterations. Speaker identity comes from a 3-second reference clip. Gradients never flow between the STT backbone and TTS. After the backbone is trained, everything else is frozen and only the RNN-T prediction and joint nets are fit.
Training has two stages. CPT turns plain text into pseudo-dialogues and synthesizes two-channel audio: 95% speech-text pretraining, 5% single-turn QA. SFT uses weighted round-robin, roughly 46.8% retention, 23.8% conversational behavior, 26.0% tool calling, 3.4% safety. Tool data is not TTS of textual function-calling transcripts, because URLs and code have no spoken form. A multi-agent pipeline writes scenarios, then TTS and ASR round-trip checks. SFT online augs include early interruption (p=0.1), injected backchannels (p=0.05), and a 160 ms delay on text targets. 64 GPUs, bf16, AdamW at 5e-5.
Two inference patches sit on top. Filler speech covers tool latency. If native turn-taking fails, RNN-T heuristics force-inject BOS or EOS. Incoming audio is still transcribed during tool execution, but it does not condition generation, so barge-in is off in that window.
Tables label the system V-Model. That is VoiceChat.
On FDB 1.0, among open-weight systems it has the lowest pause takeover: 15.3% synthetic, 25.5% CANDOR. User-interruption TOR is 100%, post-interruption quality 4.33/5. Smooth-turn TOR is 81.5% at 448 ms; PersonaPlex is better there at 90.8% and 170 ms. Against Gemini Live 2.0 it wins every reported FDB 1.0 number, including 853 ms and 703 ms less latency on smooth turns and interruptions. GPT-Realtime is more restrained on pauses (1.0% synthetic TOR) but slower to take a turn (1.47 s) with interruption quality 3.85.
| System | Pause TOR (synth) ↓ | Interrupt TOR ↑ | Interrupt quality ↑ | Smooth-turn latency ↓ |
| VoiceChat | 15.3% | 100% | 4.33 | 448 ms |
| PersonaPlex | 35.8% | 95.0% | 4.29 | 170 ms |
| Gemini Live 2.0 | 25.5% | 89.1% | 3.38 | 1.301 s |
| GPT-Realtime | 1.0% | 97.0% | 3.85 | 1.470 s |
On FDB 1.5 user backchannels, Resume is the desired behavior. VoiceChat resumes 93% of the time and answers unnecessarily on 1% of items, matching Gemini Live 2.0 exactly. GPT-4o Realtime resumes 70% and lands in Unknown 25% of the time.
VoiceBench normalized average is 55.1, tied with Freeze-Omni at 55.2, and more than 20 points above Moshi (29.5) and PersonaPlex (30.6). Knowledge tasks are the strong suit: OpenBookQA 61.3 vs Freeze-Omni 31.0, MMSU 46.1 vs 28.1, AdvBench refusal 100%. SD-QA, open-ended quality, and IFEval are weaker. Cascaded DuplexCascade scores 65.4. MiniCPM-o 4.5 reports 76.1 under a different open-ended judge (GPT-5.4), so that number is not a matched comparison.
FDB 3.0 uses real speech, disfluencies, and chained APIs. Tool-selection F1 is 82.5%, above Gemini Live 2.5 (78.6%) and 3.1 (81.7%). Argument accuracy is 42.2% and Pass@1 is 33.0%; the Gemini Live numbers sit near 59% and 49% to 54%. Routing is ahead of argument filling and end-to-end execution.
Appendix ASR on OpenASR: mean WER 9.02% at 80 ms chunks, 8.28% at 160 ms. VoiceChat-TTS on LibriTTS unseen speakers, turn 1: 2.00% WER, 4.380 SQuIM-MOS, cleaner than Audio Flamingo 3-Chat (4.51%, 3.600). By turn 4, WER is 2.20% and speaker similarity drops from 0.757 to 0.685. Zero-shot voice identity drifts. On one H100 80 GB with four concurrent streams, p95 latency is 118 ms per 160 ms audio chunk, about 1.36× real time.
This is the first fully open full-duplex speech model with general tool calling, and it keeps turn-taking intact. For anyone building a voice agent in-house, the path is shorter than ASR plus LLM plus TTS plus a separate VAD. Weights and the training recipe are public.
It is not the best at everything. Intelligence matches Freeze-Omni. Smooth-turn speed loses to PersonaPlex. Argument accuracy and Pass@1 trail Gemini Live by a wide margin. Read it as a systems paper you can reproduce: parallel specialized streams can put tools on a full-duplex timeline, tool routing already competes with closed APIs, argument grounding does not.
The authors list a long set. Audio context is about two minutes; longer conversations may not stick. Training trades knowledge against naturalness, transcription, turn-taking, and tools, so knowledge, instruction following, reasoning, and safety may lag the 9B text backbone. Keep at most five tools per session. Simultaneous multi-tool calls are unreliable. The model can skip calls, pick the wrong tool, invent arguments, or answer from internal knowledge when a tool should run. Long tool responses delay the next utterance. Barge-in is unavailable during tool execution. Strong noise, reverb, and competing speech still hurt.
The evals also leave gaps. FDB 1.0 and 1.5 use pre-recorded user audio, not live two-way talk. VoiceBench is single-turn intelligence. Tool data is synthetic scenarios plus TTS, which may not match real API traffic. Tables call the system V-Model while the title says NemotronLabs VoiceChat. CPT and SFT lean hard on TTS-rendered dialogue; how well that covers real pauses, overlap, and accent is not broken out.