Diarization models tested on 15 clinical talks: Pyannote 3 lowest error, Nemotron 3 fastest
MajesticAd2862 · reddit · 2026-09-26
A developer at Omi benchmarked leading speaker diarization models on 15 mock doctor–patient consultations (2.4 hours from PriMock57), with audio, references and scorers released publicly.
Batch results (DER, lower is better)
- Pyannote Precision-3: 2.891% (API, 18.9s/recording)
- Nemotron 3: 4.803%, fastest local model (0.688s/recording on one NVIDIA L4)
- Pyannote Community-1: 6.620%; Sortformer v1: 6.778%; v2.1: 7.974%
- VibeVoice-ASR: 8.233% (123s); Meta Muse Voice Transcribe: 13.042% (tested in 10-minute clips, not whole-recording)
Streaming: Pyannote live API 3.959%, Nemotron 3 4.971%, Sortformer v2.1 6.958%, VibeVoice 1.5B/7B both above 17%.
Same weights, better runtime: without changing weights, their proprietary runtime cut Nemotron's DER from 4.803% to 3.174% (34% lower error, 2.13× faster) and Community-1's from 6.620% to 5.435% (24× faster). The author cautions the eval set is small and runtime settings were tuned on it.
More from Models
- Lukas Kaiser explains the recipe: distilled reasoning traces plus RL — lukaszkaiser · 2026-09-26
- LLM long arithmetic allows zero error: one wrong token fails the whole evaluation — ctjlewis · 2026-09-26
- OpenRouter: keeping up with new model launches has become a full-time job — jeff_weinstein · 2026-09-26
- Opus 5.5 writes shorter sentences and 95% fewer em dashes, but hedges rise 97% — arena · 2026-09-26
- Frontier models have 'memoized' 2x2-digit multiplication, no intermediate steps needed — ctjlewis · 2026-09-26
- DL Weekly #474: GPT-6 Sol and Luna, AI Evals guide, and an RRSI paper — dl_weekly · 2026-09-26