Beyond the Benchmark: Conditioning, Domains and Failure Modes of Qwen3-TTS
Séverin Baroudi, Yanis Labrak, Petr Motlicek, Hervé Bredin, Ricard Marxer
cs.SD, cs.CL
2026-10-17
Qwen3-TTS tested across four domains: domain choice explains 4x more variance than system; a wrong word in a short ICL transcript multiplies hallucinations tenfold.
The Qwen3-TTS technical report (1.7B model) claims state-of-the-art zero-shot voice cloning: arbitrary text spoken in a voice taken from a short reference clip, no fine-tuning. It leaves three deployment questions open: which of the two cloning interfaces to use, how far the model holds up outside clean read speech, and where it fails. Standard evaluations lean on ASR word error rate, speaker similarity and predicted MOS, nearly always on read speech, and those measures saturate as systems approach human quality. This study treats the two interfaces as separate systems and tests them against Chatterbox, XTTS-v2 and F5-TTS on four domains: LibriSpeech (read), AMI (far-field meetings), IEMOCAP (emotional dialogue) and ATCO2 (air-traffic-control radio), one hour of test material each.
Main results use the shifted pairing, where the voice comes from a different utterance than the text so prosody cannot be copied, which matches deployment. Hallucinations are flagged from duration signatures alone; a manual audit gives recall 94.7% and F1 68.8%, so the flag overestimates the true rate but applies one rule to every system.
| Metric | Qwen3-TTS | Best baseline |
| Human fooling rate, AMI | 27.0% (ICL) / 35.6% (X-Vector) | Chatterbox 9.0% |
| Overall S, mean of 4 domains | 33.4 / 34.3 | Chatterbox 31.9 |
| Hallucination rate, standard setup | 4.1% (ICL) / 2.5% (X-Vector) | XTTS-v2 18.5% |
| Mean intelligibility score | 42.1 / 43.5 | XTTS-v2 37.6 |
The two interfaces split the work. ICL, which clones by continuing from reference audio plus its transcript, matches timbre more closely on clean references (speaker score 80.2 vs 61.5 on LibriSpeech); X-Vector, which conditions on a speaker embedding alone, wins prosody on every corpus (by 2.6 to 11.3 points). The clearest failure mode is ICL with a wrong reference transcript: hallucination rises from 1.1% to 9.0% under a lightly altered word and 14.4% under a replaced word, and on IEMOCAP from 0.8% to 21.2%. Transcript length is the driver: word corruption triggers hallucination in 65% of three-word transcripts but under 20% once the transcript passes eight words. The injected distractor is spoken aloud in 7.0% of utterances, landing in the first third of the output 96.6% of the time.
The variance decomposition carries the headline number: corpus explains 63% of score variance, system only 15%. LibriSpeech averages S 43.4 while ATCO2 drops to 18.5, with intelligibility collapsing to 14.7 against 55.2.
AASIST separates Qwen3-TTS from genuine speech in one of eight cells (ICL on LibriSpeech, 39.0% detection against 19.0% false alarms); on AMI it catches one X-Vector clone in a hundred while rejecting 17% of the real recordings. Detection and perceived realism do not track each other: XTTS-v2 is flagged 86-91% across all four corpora yet fools only 6% of listeners, and on ATCO2 the detector labels 99% of genuine recordings as spoof, more often than the clones themselves (75-95%).
For deployment this reads as a selection rule: use ICL when the reference is clean, its transcript verified and over ten words; use X-Vector for short, noisy or untranscribed references, and whenever timing matters. Domain explains four times more variance than system choice, so a small in-domain 2AFC check pays off more than switching models. The evaluation message is just as blunt: the overall score correlates with fooling at only ρ = 0.20, speaker similarity correlates negatively (−0.70; F5-TTS has the most similar voice and the least convincing clones), and the LLM judge agrees with humans on at most 16.7% of the clips that fooled them. No proxy replaces a small human test in the target domain. On safety, an off-the-shelf countermeasure offers essentially no protection against Qwen3-TTS and none on ATC radio, which makes the EU AI Act's disclosure requirement more dependable than post-hoc detection.
The human panel is three entry-level annotators at one site, and the authors concede it cannot settle the 35.6% vs 27.0% gap between interfaces. Human data exists only for AMI; the other three domains rest on an LLM judge the paper itself shows to be unstable, flipping on up to 43% of pairs when presentation order changes. Metric selection rests on four correlation points, and the equal weighting of the four dimensions was never calibrated. The duration-based hallucination flag overestimates, and the anti-spoofing verdict covers only the unrecalibrated AASIST, so a channel-matched detector might still recover ATC. Everything is English, one hour per corpus, zero fine-tuning.