Who Said What, and Will It Be Remembered? Evaluating Persistent Speaker Attribution Across Meetings
Shantanu Vispute, Aditya Mishra, Siddhartha Saxena
cs.SD, cs.AI
2026-09-30
Proposes SI-cpWER, attribution scored under one corpus-global speaker map: per-meeting leader ElevenLabs adds ~30 points on CHiME-6; ThyVoice leads the panel at 47.13.
Systems that turn meetings into long-term memory need transcripts that preserve both the words and stable speaker identities. The same person has to keep one ID across recordings, or "who committed to what last week" cannot be answered by linking facts between meetings. No standard metric checks this. WER ignores speakers. cpWER, the common speaker-attributed metric, remaps anonymous speakers independently inside each recording, so the same person can wear a different label every meeting and the metric never notices. The failure cost is asymmetric too: a dropped word lowers recall, but a word pinned to the wrong speaker sends a commitment or action item to the wrong person.
One part protocol, one part reference system.
The metric, SI-cpWER, computes cpWER under a single corpus-global one-to-one speaker map, solved with a Hungarian assignment on word-edit cost. The identity namespace starts empty and recordings arrive in a fixed order, so no pre-enrolled roster exists and speakers must be discovered online. The gap between cpWER and SI-cpWER is the net error the identity layer adds, or repairs.
Three conditions: all 129 CHiME-8 NOTSOFAR meetings as clean single-channel audio; a fixed noise-augmented copy (mean SNR 10.1 dB); and CHiME-6, two far-field dinner-party sessions cut into roughly 300-second chunks processed serially against one shared voiceprint bank, with both sessions' strangers deliberately enrolled in the same bank. Five commercial cascades (ElevenLabs, PyannoteAI, AssemblyAI, Deepgram, OpenAI) and two open baselines (SE-DiCoW, WhisperX) all resolve their local speaker clusters through the same pyannote voiceprint backend, so only their diarization and recognition differ.
ThyVoice, the authors' end-to-end entry, is built on gated speaker evidence. DiariZen md-v2 produces overlap-aware speaker activity; MossFormer2 separates bounded two-speaker regions; a separated stream re-enters a speaker's pool only if its WeSpeaker embedding cosine-matches clean evidence above 0.25. Each speaker's clean segments are stitched into one stream for Qwen3-ASR. Matching reuses an identity at cosine similarity of at least 0.5, while updating a stored voiceprint requires 0.7 plus clean or confidently separated audio; raw overlap is never enrolled. The reasoning is plain: mixed speech written into a voiceprint store as single-speaker evidence is the main source of identity drift, so enrollment passes through a gate.
| System | SI-cpWER mean | cpWER mean |
| ThyVoice | 47.13 | 46.90 |
| ElevenLabs | 54.75 | 40.47 |
| SE-DiCoW | 55.22 | 41.87 |
| PyannoteAI | 59.82 | 49.55 |
| AssemblyAI | 61.32 | 50.79 |
| WhisperX | 67.27 | 59.21 |
| OpenAI | 72.52 | 60.84 |
| Deepgram | 75.97 | 72.70 |
ElevenLabs holds the best commercial cpWER in every condition and the best WER (mean 33.73), yet falls behind ThyVoice on persistent attribution. On CHiME-6 its SI-cpWER sits about 30 points above its per-recording cpWER (74.22 vs 44.25); ThyVoice's gap is 1.5 points. The NOTSOFAR reference contains 12 speakers; AssemblyAI resolves 75 identities on clean audio and 101 under noise, SE-DiCoW 71 and 124, ThyVoice 12 and 12.
The overlap-policy ablation on 20 meetings isolates the mechanism. Separation with rematching scores 32.73 SI-cpWER; discarding overlap costs 54.12; keeping mixtures for ASR while enrolling identity from clean audio lands at 41.42; letting mixtures into the voiceprint store collapses to 67.76. The last two differ by 26 points at nearly identical cpWER, which pins the failure on contaminated identity evidence rather than transcription. ThyVoice's DiariZen component also posts the lowest DER/JER in every condition (mean 38.69/43.18). Leading at the word layer and the local-speaker layer does not deliver persistent identity; that inversion is the central result.
Anyone wiring transcription APIs into memory is selecting on per-recording numbers that do not predict cross-meeting behavior. Misattribution is more poisonous than omission for downstream retrieval, since decisions and action items land on the wrong person. Most commercial APIs expose one speaker label per word with no simultaneous-speaker surface (overlap recall 0.0 on the exposed surface in the diagnostics, versus 78.5 for DiariZen on clean audio), so a downstream identity layer cannot even see what to reject. The gated-evidence principle, treating transcription audio and identity evidence as separate selection decisions, transfers to any pipeline. Being honest about scale: every ThyVoice component is off the shelf, and the contribution is the protocol plus the composition.
The authors state the main ones. NOTSOFAR meetings are single sessions; persistence is simulated by serial processing in one fixed order against a shared voiceprint store, not observed across days, and a different order could shift results. Diagnostics score heterogeneous exposed surfaces rather than one uniform interface. Evaluation is English, single-channel; the Noisy condition is deterministic augmentation, and CHiME-6 contributes two sessions. Ablations cover 20 meetings and are directional only. Task-level harm from misattribution is unmeasured, left as future work.
A structural concern from this reading: ThyVoice runs its own gated voiceprint store while every competitor gets the same bolt-on pyannote identity layer with stitched-30s enrollment. Some of the margin may come from how well that backend fits each provider's output surface, not from each provider's latent persistent capability.