NVIDIA's Streaming TTS Synthesizes a Frame in 9.6ms and Takes Interruptions Without Losing Context

VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents

Edresson Casanova, Jaehyeon Kim, Mariana Graterol Fuenmayor, Shehzeen Hussain, Viacheslav Klimkov, Valentin Mendelev, Mikyas Desta, Paarth Neekhara, Piotr Zelasko, Chen Chen, Elena Rastorgueva, Ke Hu, Ankita Pasad, Xuesong Yang, Aya Alja'fari, Rajarshi Roy, Rohan Badlani, Jason Roche, Jason Li, Zhehuai Chen

eess.AS, cs.CL

2026-08-14

NVIDIA open-sources a 977M streaming TTS that consumes LLM token streams continuously, stops within 89.9ms of a user barge-in without resetting the KV cache, and beats Qwen3-TTS per-frame latency by 2.1x.

What problem this solves

The hard part of natural spoken interaction is not sounding good, it is staying responsive. Most speech language models are turn-based: the user finishes, the model thinks, the model speaks, with an awkward gap in between, and a user who wants to barge in has to wait out the current turn. Recent duplex speech-to-speech models fold ASR, understanding, and synthesis into one network, which cuts latency but forces three objectives into a joint optimization that often costs speech quality.

The NVIDIA team take the opposite bet: keep the modular pipeline, but make the TTS stage itself a continuously-on streaming model that consumes LLM text tokens as they arrive instead of waiting for a full sentence.

Method

The model totals 977M parameters: a 778M streaming TTS trunk built on Gemma 3, plus a 199M neural codec. The acoustic side runs at 12.5 frames per second, each frame covering 80 ms of audio, quantized with a 31-codebook RVQ.

Key design choices:

• Direct consumption of LLM token streams. Text uses the Nemotron Nano 2 subword tokenizer, measured at 4.16 tokens per second on LibriTTS test-clean, roughly one text token per three acoustic frames. The text stream leads the audio stream by one token, so the model sees at least two subwords before it has to speak, a minimal amount of linguistic look-ahead.
• Character-Aware Subword Encoder. LLM vocabularies and TTS training corpora mismatch, so the LLM emits subwords the TTS has never seen. Each subword is decomposed into characters, run through a shallow Transformer encoder, and average-pooled back into one embedding. Unseen words can still be sounded out.
• Control tokens. A BOS token marks the start of each assistant turn; an interruption token marks where a user barge-in should stop speech and transition to silence. The model is explicitly trained to emit silence when no text input is available, so it can sit on the conversation timeline continuously without start/stop scheduling.
• Interruption without KV cache reset. Context survives a barge-in and generation picks up from where it left off, which is the substantive engineering difference from most streaming TTS.

Results

Against Qwen3-TTS-12Hz-1.7B, Audio Flamingo 3-Chat, and Chatterbox-TTS (unseen speakers, single turn):

Single-turn intelligibility sits slightly behind Qwen3-TTS and clearly ahead of the other streaming model, Audio Flamingo 3-Chat. Multi-turn stability is the point: WER drifts only from 2.00% to 2.20% across turns one through four.

On interruption handling (200 controlled barge-in examples): with Force Silence enabled, IOR@320ms reaches 100%, mean stop latency 89.9 ms, speech leakage in the first second after interruption 55.8 ms, and the post-interruption turn lands a CER of 0.095% without any KV cache reset.

Latency (RTX A6000, concurrency 1): 9.62 ms to produce one acoustic frame versus 20.34 ms for Qwen3-TTS-12Hz, a 2.1x speedup. Since a frame only covers 80 ms of audio, synthesis comfortably outruns playback.

A final comparison holds PersonaPlex's predicted text-token stream fixed and swaps only the synthesizer: CER drops from 4.06% to 2.05%, WER from 5.00% to 2.42%, Squim-MOS rises from 4.094 to 4.292. Same text, different voice, that much difference.

Why it matters

Latency in voice agents accumulates along the chain, with every stage waiting for the previous one to finish. This paper shows the TTS stage can be made stream-in, stream-out, interruptible at any moment, and recoverable without losing context, all without speech quality collapsing. For teams building voice assistants and companion agents, it is a path to duplex interaction that does not require going end-to-end S2S. The weights ship inside the open NVIDIA-NemotronLabs-VoiceChat-11B release.

Limitations

The authors' own list: no user-audio conditioning, so the model cannot hear the user's acoustic cues and adapt its prosody; and no systematic component-wise ablation, since the architectural changes were complementary and only preliminarily shown to be cumulative. Continuous duplex TTS also lacks standardized benchmarks, so long-horizon generation and recovery across repeated interruptions go unmeasured, which the paper flags as future work.

One number deserves caution: unseen-speaker SECS (speaker similarity) drops from 0.757 at turn one to 0.685 at turn four, so zero-shot voice identity drifts over long continuous generation. Squim-MOS is a non-intrusive estimate, not a human rating, and its absolute values should not be quoted as MOS.

Terms

• barge-in: the user interrupting while the AI is speaking; a hard real-time requirement for conversational speech systems
• duplex: a conversation mode where both sides talk simultaneously, like a phone call rather than a walkie-talkie
• RVQ codebook: residual vector quantization, the structure neural codecs use to compress audio into multiple discrete token books
• KV cache: the key/value cache kept during Transformer inference; preserving it means an interruption does not force recomputing the whole history
• Squim-MOS: a model-estimated speech quality score that approximates human MOS ratings without running listeners
• ITL: inter-token latency, the time to emit one frame of audio tokens

Source

• arXiv 2608.13831v1
• PDF
• Hugging Face

What people are saying

• VoiceChat-TTS: Low-Latency Streamable TTS Supporting Mid-Utterance Barge-In rdesh26

Related papers

• NAPE-L ties SSLAM at 50.2 AudioSet mAP with only next-patch embedding prediction
• One stem at a time: flow-matching music mixing wins 5 of 6 blind listening tests
• WanSong: pure diffusion turns out 5-minute songs with split vocal/BGM stems in one pass
• Test-time beam search cuts RVQ codec quantization error without retraining EnCodec or HiFi-Codec
• Free BM25 Beats an 8B Multilingual Embedder on Greek RAG. A Fine-tuned 1B Is What Reverses It.
• Modelling modulation FX with differentiable DSP: low-frequency weighting fixes delay-time learning

All paper explainers