Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems
Taejin Park, Ivan Medennikov, Kunal Dhawan, Weiqing Wang, He Huang, Nithin Rao Koluguri, Krishna C. Puvvada, Jagadeesh Balam, Boris Ginsburg
ICML 2025 Proceedings of the
eess.AS, cs.CL, cs.LG, cs.SD
2024-09-11
Sortformer ranks speakers by arrival time and injects them into ASR via sine kernels. Hybrid DER is 14.76 on DIHARD III (≤4 speakers); Canary 2-mix WER drops from 6.55 to 4.61.
Multi-talker ASR has a bookkeeping problem. Diarization says who spoke when; ASR says which words were said. Both sides hit a permutation: the model may call two people speaker 0 and speaker 1 while the labels are flipped. The usual fix is permutation-invariant loss, searching over speaker permutations at train time. That works for diarization. It becomes awkward once text tokens enter the same graph, because the ASR head then needs its own PIT, and multi-task training gets messier. Fully end-to-end speaker-attributed ASR also starves for annotated multi-talker data and often trails cascaded systems.
NVIDIA moves the permutation out of the loss and into a data convention: rank speakers by the time each person first starts talking, and make the model emit that same order. Once the order is shared, multi-talker ASR can train with ordinary token cross-entropy, almost like single-speaker ASR.
Sortformer is an encoder-only diarizer. A 115M NEST Fast-Conformer feeds 18 Transformer layers (hidden size 192) with four sigmoid heads, so at most four speakers; 123M parameters in total. Presence of each speaker is a multi-label problem, hence sigmoid rather than softmax.
Sort Loss sorts the ground-truth speaker rows by arrival time, then applies binary cross-entropy. There is no search over K! permutations. Arrival-time estimates can be wrong, so a hybrid loss mixes Sort Loss and PIL with weight α; the paper uses α=0.5.
The Transformer needs positional embeddings to learn this order. Without them, multi-head self-attention is permutation-equivariant and cannot tell who spoke first.
The bridge into ASR is a sinusoidal speaker kernel. Sortformer's frame-level posteriors are multiplied by a bank of sine kernels and added onto L2-normalized ASR encoder states. Transcripts insert sorted tokens such as spk0 and spk1 (Sorted Serialized Transcript). Word-level objectives put a speaker token before every word; segment-level objectives look closer to serialized output training. The ASR loss stays ordinary cross-entropy. Sortformer can be frozen or jointly tuned; for the 1B Canary, only adapters are trained so the original ASR skill is kept.
On standalone diarization, one model is evaluated on three sets with no per-dataset fine-tuning. Hybrid loss plus timestamp post-processing:
| System | DIHARD III ≤4 spk, 0s collar | CALLHOME 2/3/4 spk, 0.25s | CH109 2-spk |
| Sortformer hybrid | 14.76 | 5.87 / 8.46 / 12.59 | 6.86 |
| PIL only | 17.04 | 6.94 / 10.30 / 17.52 | 6.89 |
| EEND-EDA | 15.55 | 7.83 / 12.29 / 17.59 | not reported |
Sort Loss alone is already close to PIL; combining them is what pulls ahead. Against PIL-only, hybrid DER on DIHARD III drops from 17.04 to 14.76, and CALLHOME 4-speaker from 17.52 to 12.59. EEND-GLA-Large still reports 13.64 on DIHARD III, with a clustering stage, so the comparison is not strictly end-to-end.
Plugged into Canary-170M, the stack is 293M (Sortformer-MS-Canary). On LibriSpeechMix:
| System | 1-mix | 2-mix | 3-mix |
| Canary-170M | 2.19 | 21.37 | 48.71 |
| MS-Canary, no speaker supervision | 2.74 | 6.55 | 12.14 |
| Sortformer-MS-Canary | 2.26 | 4.61 | 9.05 |
| SOT-ASR-SQR | 4.2 | 8.7 | 20.2 |
| MT-LLM, 8.4B | 2.3 | 5.2 | 10.2 |
Against MS-Canary, relative WER drops about 30% on 2-mix and 25% on 3-mix. On real AMI-test (≤4 speakers), a 1B Canary with 256-d adapters and word-level speaker tokens reaches 18.04% WER and 26.71% cpWER; the 170M model without speaker supervision is 19.67%/32.94%. Adding Sortformer increases ASR processing time by 0.78%.
The usable takeaway is the training interface. Multi-talker ASR becomes "insert sorted speaker tokens into the transcript and train with cross-entropy." No separate PIT for speakers, and no word-level timestamps. Code and weights ship in NVIDIA NeMo.
This is an integration paper, not a race to shave more DER. Teams already on Canary or Fast-Conformer get a cheap adapter path. Teams hand-writing PIT for multi-talker ASR get a simpler recipe. Whether the same sine-kernel channel drops into a multimodal LLM is still a claim without an experiment.
The hard cap is four speakers, and long sessions are sliced to 90 seconds. DIHARD III numbers cover only the ≤4-speaker subset, so this is a different track from systems that handle arbitrary speaker counts. CH109 slightly overlaps CALLHOME part1; the authors call the leak minor and do not report a cleaned split.
On real meetings, cpWER stays well above WER, so speaker swaps remain. Jointly fine-tuning Sortformer is worse than freezing it: AMI cpWER rises from 28.17 to 32.74. LibriSpeechMix is synthetic overlap. The paper does not show the sine kernel attaching cleanly to an arbitrary multimodal LLM.