Whisper gets only 34% of dictated phone numbers right in German and French, benchmark finds
Gjuesev · reddit · 2026-10-15
A new benchmark, VoxEntity-EU, tests whether speech systems capture dictated names, phone numbers, emails, dates, addresses and reference codes exactly in German and French — one wrong character fails the slot.
Results: Whisper large-v3-turbo scores 34.2% exact slot accuracy and Voxtral Mini 3B 37.1% (statistically indistinguishable, published as a null result), while feeding the same correct text to the extractor hits 90.8% — the ASR is the bottleneck. Emails land at 0–20%, phone numbers at 10–25%; transcripts read perfectly, so WER barely notices, yet every downstream form stores a wrong number. A simulated 8kHz phone line costs Whisper another 5 points.
Honest caveats: speech is synthetic (Qwen3-TTS); with Piper TTS Whisper gains 10 points, so numbers are released per TTS family. Everything rescores on CPU offline, with a browser to listen to each failure. Dataset, code, and reports are open on Hugging Face and GitHub.
More from Research
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23
- Perceptic launches PercepticOS, an AI workspace for 'scientific superintelligence' — nathanbenaich · 2026-10-23
- EPFL's Bosselut to talk long context as test-time learning at ELLIS seminar — andre_t_martins · 2026-10-23