AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
Ziyang Ma, Zhikang Niu, Wenming Tu, Tianrui Wang, Ruiqi Yan, Junxi Liu, Yanru Huo, Nickk Huang, Yang Liu, Qicong Xie, Zeyu Xie, Hui Wang, Haitao Li, Zixuan Jiang, Yalin Li, Jie Fang, Yifan Duan, Zeyue Tian, Guangzheng Li, Haina Zhu, Shuyi Wang, Jinwen Wang, Mingyu Cui, Tian Tan, Auden, Sen Liang, Steve Yves, Shan Yang, Liefeng Bo, Zilong Zheng, Kai Yu, Eng-Siong Chng, Xie Chen
cs.SD, cs.CL, cs.MM
2026-09-08
AuK unifies generation and five editing families on 3.03B instruction pairs, with 2.65% Seed-TTS-Eval error; 4-step AuK-Flash is 4.5× faster.
Speech systems already cover zero-shot cloning, instruction-conditioned TTS, local word edits, emotion or accent changes, denoising, and speaker separation. Those skills still live in separate models. A real request often chains several of them: invent a voice from a description, rewrite half a sentence, pull the accent back, then strip background noise. Swapping models and APIs fragments the product and duplicates training.
Unification is hard because the output contracts differ. Generation creates a waveform from scratch. Content editing may touch only a span. Paralinguistic and acoustic edits must keep the words. Enhancement and separation keep only what the instruction names. Conditioning is split as well: some tasks are text-only, others need joint reasoning over an instruction and reference audio. Metrics split the same way. WER and speaker similarity scale; open-ended edits also hinge on naturalness, edit strength, and whether untouched attributes stayed put. AuK folds five task families into one interface: a natural-language instruction plus optional audio context, mapped to a target waveform.
The pre-training corpus is built as five families, about 3.03 billion instruction-audio instances and 1.95 million hours of effective supervision. In the joint stage, generation takes 28.10% of each batch, content editing 23.02%, enhancement and separation 23.17%, paralinguistic editing 21.75%, and acoustic editing only 3.96%. Zero-shot TTS is trained as transcript-free, cross-utterance in-context learning: distinct utterances from the same speaker are paired, the prompt is waveform-only, and inference does not need ASR on the reference. Instruct TTS uses Qwen3-Omni captions. Most editing pairs are synthetic. Acoustic attributes come from deterministic signal processing. Emotion, timbre, de-accenting, nonverbal events, and content infill are generated with Qwen3-TTS, IndexTTS2, CosyVoice2, OmniVoice, F5-TTS, and YingMusic-Singer-Plus. Enhancement and separation mix noise, reverb, channel damage, and overlapping speakers so the model removes only the layer named in the instruction.
Three frozen or trained pieces sit in the stack. Semantic conditioning comes from frozen Qwen2.5-Omni: layer-wise hidden states are LayerNormed and combined with learned scalar weights, for text-only and audio-conditioned inputs alike. Acoustic conditioning comes from AuK-VAE, which compresses 24 kHz audio to 64-dim latents at 50 Hz and was trained for 1.24 million updates on about 3 million hours of speech, music, and general audio at a 6:3:1 mix. The generator is a FLUX-style rectified-flow Transformer with about 1.5 billion parameters: 10 dual-stream MMDiT blocks run joint attention with separate residual paths, then 20 single-stream DiT blocks predict the target-latent velocity. The MLLM and VAE stay frozen; only the backbone and layer-fusion weights are trained.
Training has four stages. A 50k-update generation-only warmup is followed by 600k joint flow-matching updates on 256 GPUs with ZeRO-2. Post-training splits. Open-ended editing has no broad reward model, so annotators score 10 or 20 candidates per instruction on a three-level scale; after filtering, 818 groups and 9,080 ratings remain. A flow-based DPO score plus LiPO's listwise loss runs for 104 updates. Generation uses Flow-GRPO for 500 updates: ASR and speaker cosine for zero-shot TTS, plus a Qwen2.5-Omni-7B style-consistency judge for instruct TTS. Distillation starts from consistency training, then task-routed Decoupled DMD. Separation examples are pulled out of DMD and trained with clean-latent regression so failed separations do not get pulled back toward the mixture by the teacher field. AuK-Flash then runs 4 steps with no CFG and is 4.5× faster in wall clock than the 32-NFE, CFG=2.0 teacher under matched conditions.
Free-form requests still go through a Prompt Enhancer: ASR, task routing, snapping rate and loudness onto discrete training bins, and duration estimation. Without that layer, unconstrained wording does not invoke the trained skills reliably.
AuK-VAE leads PESQ, STOI, Mel Dist, and STFT on speech, AudioSet, and MUSDB18-HQ. Speech PESQ on Seed-TTS-Eval is 4.143 against 3.633 for MiniMax-H3-AudioVAE.
Zero-shot TTS on Seed-TTS-Eval, mean of three runs:
| Model | Avg error ↓ | Avg SIM ↑ |
| Qwen3-TTS | 3.07% | 0.745 |
| Seed-TTS | 3.65% | 0.778 |
| VoxCPM2 | 3.65% | 0.767 |
| AuK-Flash | 2.85% | 0.790 |
| AuK | 2.65% | 0.795 |
English WER is 1.02% and Chinese hard-set error is 5.91%, both the lowest in the table. On InstructTTSEval, Chinese DSD is 83.37%, 2.27 points above Qwen3-TTS-VD. Flash ties Qwen3-TTS-VD on English DSD at 82.40%. Chinese APS is 83.28 versus 85.20 for Qwen3-TTS-VD; English APS and role-play are not a clean sweep.
MMAE-Speech is reported with the Prompt Enhancer on. IFR/CR reach 48.23%/88.11%, +4.71 and +10.84 points over Step-Audio-EditX. Flash EMR is 13.85% versus 7.04% for Ming-UniAudio, still only a little over one in ten samples passing every rubric at once. On SpeechEditBench, content editing is 91.83% versus 76.46% for Ming-UniAudio, and prosody 71.33% versus 26.50%. Emotion editing is 9.94%; every system is weak there. On Ming-Freeform Full, average WER drops from 10.46% to 3.09% in Chinese and from 14.28% to 3.96% in English.
Signal-level tasks are competitive, not dominant. DNS Challenge dWER is 2.66% versus 3.31% for RE-USE. On CHiME-4, Flash posts 7.84% WER and 3.91 UTMOS. On Libri2Mix, AuK's 9.12% WER slightly beats the specialist MossFormer2-SS at 9.34%, with SIM tied at 0.96. Flash often wins UTMOS; the full model is steadier on content accuracy.
Weights and code are released under Tencent Hunyuan. This is an open recipe that puts TTS, lexical edits, paralinguistic changes, and restoration on one backbone. The 1.5B flow backbone plus a frozen Omni encoder is more reproducible than training a giant audio LLM from scratch. The distillation detail that matters in practice is the separation-task route: uniform DMD hurt source separation, so those examples were supervised separately.
For product work, the interface and Flash are the usable parts. One instruction-plus-optional-reference path covers cloning and editing. Four CFG-free steps are closer to deployable latency. Transcript-free cloning also drops the extra ASR at serving time.
This is still incremental unification, not a new generative primitive. Editing supervision is largely distilled from F5-TTS, IndexTTS2, and similar teachers, so the ceiling tracks those distributions. Acoustic editing is only 3.96% of the joint batch.
The paper is blunt: unconstrained editing requests are not grounded natively. The Prompt Enhancer still has to classify the task and rewrite the prompt. An agent-style SFT pipeline adapted from Audio-Oscar made instructions more diverse and still failed to generalize. Having a skill and invoking it from free language remain different problems.
Preference data is 818 groups. Alignment coverage for open-ended edits is thin, which matches an EMR of 13.85%. Emotion-edit success in the single digits undercuts any claim that one model edits everything well.
The synthetic chain is long. Content infill uses F5-TTS, lyric edits use YingMusic-Singer-Plus, de-accenting uses CosyVoice2 and OmniVoice. The filter is ASR WER, so unnatural prosody and near-miss semantics can enter the set systematically. Cross-lingual de-accenting and generating whisper from text after training only whisper edits are qualitative notes, not benchmark results.
Relative error on speaking rate and loudness is not better than Ming-UniAudio, and sometimes worse. Average WER and SIM in the summary table can look like a win while control precision does not. DNSMOS-OVRL on DNS Challenge is 3.35 versus 3.41 for AnyEnhance. On pure signal metrics, specialists still take first place often enough that "competitive" is the right word.