Confucius4-R2T2: A Stable-Prefix Streaming ASR Model for Real Real-Time Transcription
Daniel Huang, Qi He, Shaoxiong Duan, Yingjie Li, Qiao Cheng, Yitao Duan
cs.CL, cs.AI, cs.SD
2026-11-04
Post-trained from Qwen3-ASR-1.7B with Longest Stable Prefix labels, R2T2 streams text that is never revised: 5.76 macro error on 13 EN/CN sets, beating all commercial APIs tested.
Streaming ASR output is increasingly consumed by machines. Live captions, simultaneous translation, and voice agents all act on each token the moment it appears, so a token that gets revised later means wasted work downstream, or for an agent, a wrong action. Conventional streaming recognizers revise constantly: as audio accumulates, earlier words get rewritten, a failure mode the literature calls flicker. Feed a full-context model sliced audio progressively and the count of correct tokens does not even grow monotonically, because each new tail disturbs earlier output; the paper calls this residual interference. Emitted text never changing deserves the same status as accuracy and latency. Existing fixes are either inference-time commit rules (local agreement releases only the prefix shared by consecutive hypotheses; hold-n withholds the last n tokens) or indirect training (prefix training, self-training). All of them lower the revision rate. None guarantees it away.
The paper defines stability as a purely behavioral constraint (at every time, the output must be a prefix of all later outputs) and trains it in directly.
R2T2 is Qwen3-ASR-1.7B post-trained on 1,500 hours of proprietary Chinese and English audio with noise, reverberation, and language-tag-flip augmentation. The 300M-parameter audio encoder and the projector stay frozen; only the language decoder is trained. No new layers, no attention masks, no causalization. Earlier work that causalized Whisper's encoder lost accuracy above roughly 300 ms chunks; freezing keeps the bidirectional representations, and the decoder absorbs the truncated-audio mismatch instead.
The core is Longest Stable Prefix (LSP) labeling. Audio is fed to the teacher (the base model itself, self-distillation) in 80 ms increments and decoded at each step; with the full-audio output as the reference, the LSP at time t is the longest earlier output that is entirely a prefix of that reference. Three choices carry the design:
Four inference optimizations leave the model untouched: encoder KV-cache reuse over 8-second attention windows; ending the update as soon as Wait is predicted, so silent stretches cost a single pre-fill; bounded context (one retained completed window, at most 16 s of audio plus transcript); and speculative-decoding-style pre-fill verification of a revisable suffix of U tokens instead of regenerating it. U switches at inference: U=1 favors accuracy, U=0 favors latency, no retraining.
For picking a backbone, the paper proposes streamability: plot the LSP length as audio arrives and take the AUC. On 1,860 Buckeye utterances, Qwen3-ASR-1.7B ranks first at 0.493 normalized, while WhisperRT, a causally fine-tuned Whisper, scores below its own full-context base because unstable outputs shorten the LSP.
Thirteen test sets (8 English, 5 Chinese), 160 ms chunks, macro-averaged WER/CER:
| Configuration | Error (%) |
| Base, offline full-context (reference) | 5.15 |
| Base, 2 s chunks, U=5 (reference) | 5.25 |
| Base, 160 ms, unadapted | 21.01 |
| R2T2, 160 ms, U=1 | 5.76 |
| R2T2, 160 ms, U=0 | 6.40 |
| X-ASR, 200 ms | 8.97 |
| Nemotron 3.5 | 14.21 |
| Voxtral Realtime | 19.35 |
| AssemblyAI | 8.27 |
| Commercial API A / B | 6.23 / 6.01 |
R2T2 at U=1 posts the lowest error of every ranked configuration, ahead of AssemblyAI and two anonymous commercial APIs, all of which expose revisable, non-stable partials. Against the unadapted base at the same 160 ms setting it cuts error by nearly three quarters; the base at U=0 was not even scorable, failing with severe hallucination. The gap to the base model's own full-context decode is about 0.6 points, and on VoxPopuli R2T2 beats full-context, 2.29 versus 3.12. Mean emission lag is lower than all three commercial APIs in both languages; Commercial A and B are more accurate in Chinese but stabilize later. At 80 ms chunks with U=0, mean retrospective lag stays under half a second in both languages, with error rising to 6.40. Transfer holds beyond the post-training languages: R2T2 beats the base at the same setting on all 18 test sets across 10 additional languages, and leads the ranked configurations on both RealSI cohorts, long-form audio from real simultaneous interpretation.
For captions, translation, and voice agents, committed text being final removes a layer of downstream engineering: no finalize policy deciding which part of a partial hypothesis is safe, and no text jumping on screen. Weights and code are open-sourced on HuggingFace and GitHub; the decoder is 1.7B with a frozen encoder, and bounded context keeps per-update cost flat as streams grow. The recipe itself, LSP labels plus a Wait token plus a frozen encoder, is a transferable template for converting full-context models into stable streamers, and streamability offers a cheap pre-test for which backbone is worth converting before spending compute.
Stated in the paper: latency is a retrospective measure on the audio timeline, excluding execution, queueing, and transport, and the sub-1 RTF assumption is never checked with runtime measurements; whether streamability generalizes past ASR and whether a stronger teacher would help are open questions; the eight reward components get no controlled ablation. Points a careful reader adds: the 1,500-hour training set is proprietary, so reproduction depends on the released weights; LSP's conservatism biases toward late emission and the cost is only acknowledged qualitatively; API latencies are estimated from supplied audio since server-side buffering is undisclosed; flicker frequency itself is not measured, though for R2T2 stability holds by construction. How much of the 0.6-point gap to full-context is intrinsic to streaming is left unexamined.