Confucius4-R2T2: NetEase Youdao's streaming ASR never revises output and beats commercial APIs

Confucius4-R2T2: A Stable-Prefix Streaming ASR Model for Real Real-Time Transcription

Daniel Huang, Qi He, Shaoxiong Duan, Yingjie Li, Qiao Cheng, Yitao Duan

cs.CL, cs.AI, cs.SD

2026-11-04

Post-trained from Qwen3-ASR-1.7B with Longest Stable Prefix labels, R2T2 streams text that is never revised: 5.76 macro error on 13 EN/CN sets, beating all commercial APIs tested.

What problem this solves

Streaming ASR output is increasingly consumed by machines. Live captions, simultaneous translation, and voice agents all act on each token the moment it appears, so a token that gets revised later means wasted work downstream, or for an agent, a wrong action. Conventional streaming recognizers revise constantly: as audio accumulates, earlier words get rewritten, a failure mode the literature calls flicker. Feed a full-context model sliced audio progressively and the count of correct tokens does not even grow monotonically, because each new tail disturbs earlier output; the paper calls this residual interference. Emitted text never changing deserves the same status as accuracy and latency. Existing fixes are either inference-time commit rules (local agreement releases only the prefix shared by consecutive hypotheses; hold-n withholds the last n tokens) or indirect training (prefix training, self-training). All of them lower the revision rate. None guarantees it away.

The paper defines stability as a purely behavioral constraint (at every time, the output must be a prefix of all later outputs) and trains it in directly.

Method

R2T2 is Qwen3-ASR-1.7B post-trained on 1,500 hours of proprietary Chinese and English audio with noise, reverberation, and language-tag-flip augmentation. The 300M-parameter audio encoder and the projector stay frozen; only the language decoder is trained. No new layers, no attention masks, no causalization. Earlier work that causalized Whisper's encoder lost accuracy above roughly 300 ms chunks; freezing keeps the bidirectional representations, and the decoder absorbs the truncated-audio mismatch instead.

The core is Longest Stable Prefix (LSP) labeling. Audio is fed to the teacher (the base model itself, self-distillation) in 80 ms increments and decoded at each step; with the full-audio output as the reference, the LSP at time t is the longest earlier output that is entirely a prefix of that reference. Three choices carry the design:

Four inference optimizations leave the model untouched: encoder KV-cache reuse over 8-second attention windows; ending the update as soon as Wait is predicted, so silent stretches cost a single pre-fill; bounded context (one retained completed window, at most 16 s of audio plus transcript); and speculative-decoding-style pre-fill verification of a revisable suffix of U tokens instead of regenerating it. U switches at inference: U=1 favors accuracy, U=0 favors latency, no retraining.

For picking a backbone, the paper proposes streamability: plot the LSP length as audio arrives and take the AUC. On 1,860 Buckeye utterances, Qwen3-ASR-1.7B ranks first at 0.493 normalized, while WhisperRT, a causally fine-tuned Whisper, scores below its own full-context base because unstable outputs shorten the LSP.

Results

Thirteen test sets (8 English, 5 Chinese), 160 ms chunks, macro-averaged WER/CER:

ConfigurationError (%)
Base, offline full-context (reference)5.15
Base, 2 s chunks, U=5 (reference)5.25
Base, 160 ms, unadapted21.01
R2T2, 160 ms, U=15.76
R2T2, 160 ms, U=06.40
X-ASR, 200 ms8.97
Nemotron 3.514.21
Voxtral Realtime19.35
AssemblyAI8.27
Commercial API A / B6.23 / 6.01

R2T2 at U=1 posts the lowest error of every ranked configuration, ahead of AssemblyAI and two anonymous commercial APIs, all of which expose revisable, non-stable partials. Against the unadapted base at the same 160 ms setting it cuts error by nearly three quarters; the base at U=0 was not even scorable, failing with severe hallucination. The gap to the base model's own full-context decode is about 0.6 points, and on VoxPopuli R2T2 beats full-context, 2.29 versus 3.12. Mean emission lag is lower than all three commercial APIs in both languages; Commercial A and B are more accurate in Chinese but stabilize later. At 80 ms chunks with U=0, mean retrospective lag stays under half a second in both languages, with error rising to 6.40. Transfer holds beyond the post-training languages: R2T2 beats the base at the same setting on all 18 test sets across 10 additional languages, and leads the ranked configurations on both RealSI cohorts, long-form audio from real simultaneous interpretation.

Why it matters

For captions, translation, and voice agents, committed text being final removes a layer of downstream engineering: no finalize policy deciding which part of a partial hypothesis is safe, and no text jumping on screen. Weights and code are open-sourced on HuggingFace and GitHub; the decoder is 1.7B with a frozen encoder, and bounded context keeps per-update cost flat as streams grow. The recipe itself, LSP labels plus a Wait token plus a frozen encoder, is a transferable template for converting full-context models into stable streamers, and streamability offers a cheap pre-test for which backbone is worth converting before spending compute.

Limitations

Stated in the paper: latency is a retrospective measure on the audio timeline, excluding execution, queueing, and transport, and the sub-1 RTF assumption is never checked with runtime measurements; whether streamability generalizes past ASR and whether a stronger teacher would help are open questions; the eight reward components get no controlled ablation. Points a careful reader adds: the 1,500-hour training set is proprietary, so reproduction depends on the released weights; LSP's conservatism biases toward late emission and the cost is only acknowledged qualitatively; API latencies are estimated from supplied audio since server-side buffering is undisclosed; flicker frequency itself is not measured, though for R2T2 stability holds by construction. How much of the 0.6-point gap to full-context is intrinsic to streaming is left unexamined.

Terms

Source

What people are saying

Related papers

All paper explainers