Artificial Analysis Launches Pronunciation Robustness Benchmark for TTS Models

On September 22, Artificial Analysis released a new TTS Pronunciation Robustness benchmark, where human reviewers judged whether English passages were pronounced correctly — 454 sentences and 701 target words in total, covering four challenge categories: context-dependent pronunciations, abbreviation expansions, exact sequence preservation, and standalone terms. A TTS model comparison page integrating preference, speed, price, and pronunciation metrics was launched alongside it.

Confirmed

• Gemini 3.1 Flash tops the chart with 88.1% pronunciation accuracy, also leading in context-dependent pronunciations (96.4%) and abbreviation expansions (84.4%), though its abbreviation expansion score is only 0.1 percentage points ahead of SpaceXAI TTS
• Pricing: Gemini 3.1 Flash costs $18.31 per million characters; SpaceXAI TTS is priced at $15, the cheapest model scoring above 80%; Kokoro is just $0.65, the cheapest overall
• Speed: SpaceXAI TTS reaches 87.6% at 106 characters per second; Realtime TTS-2 Flash runs at 220 characters per second but only 77.8% accuracy; Kokoro is fastest but scores just 55.5%
• The largest gap among models is in exact sequence preservation, with scores ranging from 5.3% to 85.7%
• Preference and pronunciation don't align: Sonic 3.6 has the highest Elo (1276) in the Provider Voice Arena but only 74.5% pronunciation robustness; Gemini 3.1 Flash's Elo is 1201
• In results later published on the full TTS leaderboard page, xAI ranks first with 87.6% pronunciation accuracy, inconsistent with the initially announced Gemini 88.1% leaderboard

Why it matters

Preference-based Elo rankings only capture how good a voice sounds and can't expose hard failures like mispronounced abbreviations or proper names; the pronunciation robustness benchmark provides a new selection criterion for accuracy-sensitive use cases such as customer service and audiobooks, and combined with price and speed data enables direct cost-performance trade-offs.

2026-09-22 ~ 2026-09-22 · 6 related posts

• [source] New TTS pronunciation benchmark: Gemini 3.1 Flash TTS leads at 88.1% — ArtificialAnlys · 2026-09-22
• [source] TTS benchmark breakdown: exact-sequence category shows 5.3%-85.7% model spread — ArtificialAnlys · 2026-09-22
• Arena Elo vs pronunciation: top-preference Sonic 3.6 scores just 74.5% accuracy — ArtificialAnlys · 2026-09-22
• [source] TTS leaderboard: xAI hits 87.6% pronunciation accuracy, Kokoro 82M fastest at 242 chars/s — ArtificialAnlys · 2026-09-22
• TTS pronunciation benchmark pricing: Gemini leads at $18.31/1M chars, Kokoro cheapest — ArtificialAnlys · 2026-09-22
• TTS speed vs accuracy: SpaceXAI hits 87.6% at 106 chars/sec, Kokoro fastest but weakest — ArtificialAnlys · 2026-09-22