Qwen-Audio-3.1-ASR Technical Report
Chuanmeng Bian, Daren Chen, Peixin Chen, Zhigao Chen, Chong Deng, Zhiyun Fan, Zhifu Gao, Bo Gong, Qing Gu, Jiajun He, Yawei Hu, Yunjie Ji, Wei Ju, Jingbei Li, Xiangang Li, Xu Li, Zengxi Li, Zheng Li, Chengdong Liang, Baiji Liu, Yaming Liu, Ying Liu, Bin Ma, Yiping Peng, Yuezhang Peng, Zhendong Peng, Yu Pu, Yang Shi, Xin Shu, Jian Tang, Biao Tian, Peiyao Wang, Tianzi Wang, Wen Wang, Wupeng Wang, Cheng Wen, Yuzhong Wu, Zijian Xia, Yunchong Xiao, Nan Yang, Jianwei Yu, Jixing Yu, Binbin Zhang, Lei Zhang, Sitong Zhao, Guangdong Zhou, Yuan Zhou, Jianheng Zhuo
cs.SD, cs.CL
2026-10-21
Qwen-Audio-3.1-ASR cuts seven-language macro error from 22.50% to 15.09%; Next reaches 12.93% cpWER on diarized meetings.
A transcript answers what was said. Meetings also need who spoke when, and environmental sound or music never reduces to words. Qwen-Audio-3.0-ASR already covered 30 languages and 16 Chinese dialects, with conversation history, hotwords, and in-decoder polishing, but the output was flat text. Turn-taking, short backchannels, and overlap disappear once everything is one string.
Qwen-Audio-3.1-ASR, from Alibaba Token Foundry, keeps that recognizer and adds segments with speaker labels and timestamps. Captioning, temporal grounding, and question answering ship only in the Next variant. Streaming does neither.
The network is an audio encoder, an adapter, and an MoE decoder. The instruction, plus optional text, selects a transcript, a dialect translation, a polished rewrite, speaker-attributed segments, or a description or answer about the recording. An auxiliary CTC branch supplies a rough hypothesis for hotwords and long-audio context. It is not the delivered transcript.
Training has six stages. Some encoder layers start from Qwen3, then BEST-RQ pretrains on tens of millions of hours of unlabeled speech. Labeled AED training uses pseudo-transcripts from Paraformer-V2, Whisper-large-v3, and SenseVoice; that decoder is discarded. Audio-LLM pretraining uses about 20 million hours with the encoder frozen. A cooldown on about 1 million hours rebalances languages and dialects. After a VibeVoice pass, samples with WER above 0.3 against the original reference are dropped, and the new hypothesis does not replace the reference.
SFT updates the encoder and adapter, freezes the base LLM, and trains the decoder with LoRA. A dialect instruction either keeps dialect wording or translates into written Mandarin. Polishing runs in the same decode, stripping fillers, stutters, and explicit self-corrections, with no second rewrite model. Hotwords arrive as P0, high confidence, and P1, a wider pool; unsupported candidates stay out. Earlier transcript text may be prepended, and the current audio remains the evidence.
Speaker output is one segment per line. Labels are stable inside a recording, intervals may overlap, and they are not identities. Speaker-ID number tokens are 2.90% of supervised tokens but only 1.97% of unweighted loss, while timestamps contribute 49.51% of the loss, so speaker-ID loss is multiplied by 10. Timestamps snap to an 80 ms grid, matching the 12.5 Hz frames.
FunVerl then runs GRPO with asynchronous sampling and updates. Rewards target WER, DER, cpWER, and time-constrained tcpWER. Multi-teacher on-policy distillation routes transcription, diarization, and audio understanding into one student. Qwen3.5 scores text and dialogue, on prefixes the student actually sampled.
On clean Chinese and English, 3.1 is roughly level with 3.0. Published AISHELL-1 CER moves from 1.03% to 1.08%, FLEURS-en WER from 4.17% to 4.72%, and LibriSpeech-other from 2.24% to 2.19%.
Under one API setup, 3.1 is best on four of seven sets: FLEURS-zh 2.25% versus Tencent Hy-ASR-3.0-preview at 2.38%, FLEURS-en 4.72% versus Fun-ASR-Flash at 4.95%, LibriSpeech-clean 1.23% versus 1.35%, and LibriSpeech-other 2.19% versus 2.51%. Relative cuts are 5.5%, 4.6%, 8.9%, and 12.7%. Fun-ASR-Flash still leads AISHELL-1, AISHELL-2, and WeNetSpeech-net at 0.76%, 1.93%, and 3.83%, against 1.08%, 2.05%, and 4.37%.
The gain over 3.0 is multilingual. On seven East and Southeast Asian languages in GigaSpeechBench, 3.1 wins every language. Macro error is 15.09% versus 22.50%, a 32.9% relative drop. Doubao-ASR is at 22.76%, Gemini-3.1-Pro at 27.69%, GPT-4o Transcribe at 41.89%. Thai CER falls from 16.22% to 7.75%, Japanese from 28.48% to 18.47%. The eight-language Common Voice macro is 3.74%; Japanese CER falls from 5.64% to 1.40% and Korean from 3.09% to 0.73%, while Spanish and French stay better on 3.0. The nine-language FLEURS macro is 3.85%, ahead of 3.0 at 4.94% and behind Azure MAI-Transcribe-2 at 3.25%.
Public dialect macro CER is 4.55%, versus Doubao at 7.15% and Tencent at 4.61%, only 0.06 points under Tencent. On 16 internal dialect sets the macro CER is 10.38%, versus 20.23% and 17.13%, lowest on 11 subsets. Wenzhou is 19.21% versus 68.22% and 64.74%. Dialect-to-Mandarin semantic sentence accuracy is 88.03% versus 78.69% on Doubao's three documented subsets, and 89.03% versus 77.53% on Tencent's four. Nanchang moves from 57.44% to 89.26%. Cantonese is 93.55%, just under Tencent at 94.15%.
Across 15 industries, entity recall is best in nine and tied in two, from 94.9% to 99.5%. The larger gaps versus the stronger rival are 2.3, 2.2, and 2.1 points in AI, agriculture, and commerce. Tencent still leads four domains. Conditioned P0 hotword recall is 97.95% to 99.46%, best in every P0 category. Two P1 categories are not.
Macro cpWER on four meeting sets is 12.93% for Next and 14.12% for 3.1, versus MOSS-Transcribe-Diarize at 15.13% and Gemini-3.5-Transcribe at 32.48%. Next scores 81.60% on MMAU, above Gemini-3.1-Pro at 80.90%, and 82.24% on MMSU, below that model's 85.30%.
| Setting | Metric | Qwen-Audio-3.1 | Baseline |
| Seven-language macro | Error | 15.09% | 3.0 at 22.50% |
| Four-set diarization | cpWER | Next 12.93% | MOSS 15.13% |
| MMAU / MMSU | Accuracy | 81.60% / 82.24% | Gemini-3.1-Pro 80.90% / 85.30% |
| 16 internal dialects | CER | 10.38% | Doubao 20.23% |
| Unified Chinese-English | Sets won | 4/7 | Fun-ASR-Flash takes three |
The usable upgrade is East and Southeast Asian languages, Chinese dialects, and meetings that must keep speaker and time. Clean Chinese and English read-speech does not pull away from the previous generation, and Fun-ASR-Flash stays lower on three Chinese sets.
The variant decides what you can call. Speaker-attributed transcripts exist only on non-streaming Flash and Next. Audio understanding exists only on Next. Streaming emits no speaker labels and refreshes the full utterance at the endpoint. Hotwords and history are decode-time conditions. Polishing shares that same decode and is limited to Message, Flash, and Next. Keeping dialect wording and translating into Mandarin is a separate switch.
This reads as a product report. The 10x speaker-ID loss and the 80 ms time grid are among the few choices whose reason is spelled out.
The industry-entity gains are not isolated from the rest of the training changes. The report says so. Each vendor's public dialect list omits eight varieties, and neither lists Wenzhou, so the internal macro mixes coverage with accuracy. Wenzhou at 19.21%, with baselines still above 60%, is the sharpest case.
Semantic sentence accuracy ignores whether the output used the requested language form. Equivalent dialect and Mandarin wording both pass. The judge is Qwen3.7-Max. Agreement with humans on a subset is 95.35%, and the full set is still scored by an in-family model.
FLEURS-en error is 0.55 points worse than the previous generation. Multilingual gains sit in the Asian languages tested here. Spanish and French remain stronger on 3.0, and the FLEURS macro loses to Azure at 3.25%. Two P1 hotword categories miss the best conditioned recall.
The 10x speaker loss and the 80 ms grid have no absolute ablation, and 80 ms is the timing resolution. The available chapters do not separate the contribution of RL from distillation. Streaming latency and long-audio context figures are absent from the text read here. The training audio is not public, so an outside rerun has nowhere to start.