Artificial Analysis launches speaker diarization benchmark; NVIDIA Nemotron 3.5 leads at 3.3% WDER
ArtificialAnlys · x · 2026-11-14
Artificial Analysis released the Diarization Index, a new benchmark scoring speech-to-text APIs on speaker attribution using Word Diarization Error Rate (WDER): the share of words assigned to the wrong or no speaker.
- Test set: 97 conversations of 2–8 people, weighted equally across group sizes
- NVIDIA's Nemotron 3.5 Transcribe leads at 3.3% WDER
- Voxtral Small Transcribe 3 follows at 7.5%
- Tail models (e.g., ElevenLabs Sonic v5, Movia) exceed 16%, up to 20.1%
The benchmark fills a gap left by WER-only evaluations, and matters for meeting transcription and multi-speaker use cases where knowing who said what is critical.
More from Models
- Skeptic doubts Anthropic achieved large-factor bpb gains at fixed compute — teortaxesTex · 2026-11-14
- Surge AI benchmark: every Claude model, even Haiku, beats every GPT on staff-level coding — echen · 2026-11-14
- AI cracks Gödel's 1,009-page notebooks, uncovering hundreds of unknown theorems in days — stevenstrogatz · 2026-11-14
- NVIDIA lands nvfp4 in large-scale pretraining under codename "yolo bundle" — SonglinYang4 · 2026-11-14
- Delip Rao mocks vendors testing safety in production, then rolling back updates — deliprao · 2026-11-14
- Gemini 4 Hydrogen vs Claude Sonnet 5.5: faster, more token-efficient, better value — haider1 · 2026-11-14