VibeVoice-ASR-Streaming Technical Report
Yujie Tu, Zhiliang Peng, Jianwei Yu, Li Dong, Songchen Xu, Yaoyao Chang, Wenhui Wang, Zilong Wang, Zehua Wang, Yan Xia, Jiajun Zhang, Xie Chen, Furu Wei
eess.AS
2026-09-03
Microsoft turns offline VibeVoice-ASR into a chunked streaming model. The 7B system posts 24.66 mean WER across five sets, wins 12 of 13 speaker settings, and commits in 2s.
Meeting transcription has to emit both the words and the speaker while people are still talking. Offline LLM systems already fold ASR and speaker labels into one generative pass. VibeVoice-ASR, MOSS Transcribe Diarize, SoulX-Transcriber, and SpeakerLM all work that way, and all of them wait until the recording ends.
Streaming Speech-LLMs already chew audio in chunks and keep prior speech-text context, but they were built for one speaker. Streaming multi-talker systems usually bolt on a speaker branch, a speaker cache, or an online diarizer. Speaker identity is picky about history: the person in the current chunk may have first appeared minutes ago. Drop that history and you have to reintroduce an external speaker store. This report wants both streaming and "who said what" from a single model.
VibeVoice-ASR-Streaming starts from the offline VibeVoice-ASR stack. A 24 kHz waveform goes through frozen Acoustic and Semantic tokenizers, one latent frame every 133.3 ms, then into a trainable Qwen2.5 backbone. Incoming audio and already generated speaker-attributed text are interleaved. Each chunk is followed by a fixed 4-frame, 0.5 s lookahead before the model writes that chunk's text.
Speakers get ordinal labels in order of first appearance, reused later. Overlap is serialized as consecutive labeled segments, not parallel streams. Optional hotwords, names, and terms can be injected at the start of a session and stay in context.
Training has three stages. Stage 1 is offline multi-speaker recognition. Stage 2 switches the same model to the interleaved streaming format on about 420,000 hours of English and Chinese speech. Stage 3 fine-tunes on about 13,000 hours of public training splits plus synthetic meetings. The synthetic set is 50,884 recordings, 4,519.6 hours, with overlapped speakers and room impulse responses. The 15-frame (2.0 s) and 22-frame (2.9 s) configs are trained separately. Released weights are 1.5B and 7B; reported numbers use 7B with 22-frame chunks unless noted.
On recognition-only scoring, the 7B model is best on AliMeeting, AISHELL-4, and AMI-IHM. Gemini 3.5 Transcribe Live wins AMI-SDM. Five-set mean WER/CER:
| System | Five-set mean |
| VibeVoice-ASR-Streaming 7B | 24.66 |
| Gemini 3.5 Transcribe Live | 25.23 |
| GPT Realtime Whisper | 39.31 |
| GPT Live Transcribe | 40.55 |
| ElevenLabs Scribe v2 Realtime | 41.39 |
On speaker-attributed cpWER/cpCER it is best or tied-best on 12 of 13 settings. Against Azure ConversationTranscriber the four meeting sets improve by 2.39 to 12.45 points; the MLC-Challenge nine-language average falls from 27.06 to 22.75. Expected latency is 2.00 s, versus 8.21 s measured for Azure and 9.12 s for Google STT's first label (51.06 s after full-recording revision). Portuguese is the miss: Azure 36.11 cpWER, this model 38.25.
Streaming costs accuracy against the offline parent: WER/CER rises 0.75 to 3.53 points, cpWER/cpCER 5.13 to 6.67, so speaker assignment takes the larger hit. Growing the chunk from 15 to 22 frames saves 1.46 WER and 4.06 cpWER at 7B on the five-set mean. Scaling 1.5B to 7B at 22 frames saves another 12.76 cpWER. Head versus tail speaker labels are a wash (31.55 vs 31.56 mean cpWER). Lookahead from 0 to 4 frames drops mean WER from 27.18 to 24.66. On one A100, the 7B 15-frame config decodes a chunk in 146 to 208 ms, RTF at most 0.104. Short single-speaker audio is not the target: 22-frame four-set mean 5.76, level with X-ASR at 5.77.
A voice agent sitting with more than one person has to tag speakers while they talk. Interleaved generation can keep speaker identity consistent without a separate diarization stage. Weights and vLLM inference code are public, and 2 s latency at RTF 0.1 is already in the real-time band.
This is a streaming conversion of an offline model, not a from-scratch streaming system. It is useful in meetings and unremarkable on short single-speaker clips. The recognition-only mean beats Gemini by 0.57 points. The real gap is speaker labels and how soon they settle.
The authors list four. Language coverage is capped by Qwen3-ForcedAligner-0.6B, so training and eval sit at ten languages. Long overlaps degrade because the decoder must serialize them onto one stream. Released checkpoints stop at eight minutes; longer sessions are a compute cost, not an architectural wall. The 2.00 s figure is steady-state. The first packet waits for a full chunk plus lookahead: 3.5 s at 22 frames, 2.5 s at 15.
The comparison is also uneven. Azure and Google receive a language prior; Google also gets the reference speaker count. VibeVoice gets none of that. Eval recordings are clipped at 480 s, matching the release cap. The cloud APIs are scored as recognition plus speaker labels, not as full product stacks.