Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering
Ivan Medennikov, Taejin Park, Weiqing Wang, He Huang, Kunal Dhawan, Jinhan Wang, Jagadeesh Balam, Boris Ginsburg
eess.AS, cs.SD
2025-07-24
Streaming Sortformer caches arrival-ordered embeddings in AOSC, with no between-chunk permutation search. DIHARD III DER is 18.97 at 1.04s vs LS-EEND 19.61, and 19.32 at 0.32s.
Offline Sortformer attends over the full recording, then ranks speakers by first appearance. Live captions and online meetings cannot wait: audio arrives in chunks, and speaker indices drift across chunk boundaries. The offline-to-streaming gap in diarization is wider than the same gap in ASR. Prior streaming systems store past frames in a speaker-tracing buffer and recover permutations by correlation; streaming EEND-EDA keeps attractors and often sits at multi-second latency.
This paper takes Sortformer's arrival-time order into the streaming setting. If the model already emits speakers in first-come order, the cache can store embeddings in that same index order, and between-chunk permutation search can be dropped.
The backbone is still Sortformer: a NEST pre-encoder, Transformer layers, four sigmoids, at most four speakers. Two changes versus the offline paper: a 109M multilingual NEST on 128-dim Mel features, 117M parameters total; and no global feature normalization, which streaming cannot compute. The output hop is 80 ms.
The new piece is the Arrival-Order Speaker Cache (AOSC). It stores frame-level NEST embeddings, ordered by speaker index, which is arrival order. If the cache is under capacity, frames are appended as-is. Once full, frames are scored from the model's past predictions and only high-scoring frames are kept. Scores favor frames that look like speaker i and unlike the others; silence frames are replaced by a mean silence embedding; newly arrived frames get a small bonus; each speaker is guaranteed at least K frames. A few silence embeddings are appended after each speaker to help detect turn-taking. How many frames each speaker keeps is decided by the scores, not a fixed quota.
Short chunks lack context, so a FIFO queue sits between the cache and the current chunk, and the current chunk also keeps a little right context. The cache need not update every chunk. At each step the model sees speaker cache + FIFO + current chunk with right context. Because both the cache and the outputs follow arrival order, predictions on the cache span are discarded and only the current chunk is kept. There is no STB-style correlation permutation.
Training starts from an offline Sortformer, then fine-tunes with AOSC: 90-second clips in 15-second windows, cache size 15 seconds (188 frames). At each step the speaker order inside the cache is randomly permuted, and half the batches clip self-attention's right context to 7 frames (560 ms) so the model leans less on the future. No SpecAugment, no RIR or noise.
Evaluations use DIHARD III Eval, CALLHOME part2, and CH109, overlapping speech included. One model, no per-dataset fine-tuning. At the one-second latency band:
| System | Latency | DIHARD III all | CALLHOME all | CH109 2-spk |
| LS-EEND | 1 s | 19.61 | 12.11 | not reported |
| EEND-GLA-Large + BW-STB | 1 s | 20.73 | 14.29 | not reported |
| Streaming Sortformer-AOSC, post-proc | 1.04 s | 18.97 | 10.79 | 5.09 |
| same, 0.32 s | 0.32 s | 19.32 | 11.50 | 5.41 |
| Offline Sortformer, post-proc | unlimited | 21.39 | 11.26 | 4.86 |
At 1.04 s, DIHARD III with ≤4 speakers is 13.32 DER versus LS-EEND's 13.96. CALLHOME 4-speaker is 12.40 versus 15.30. An offline checkpoint wrapped with AOSC at inference, without fine-tuning, jumps to 29.46 DER on DIHARD III all; the cache has to be trained in.
A few results run against intuition. On 5+ speaker subsets, a 4-speaker model still tracks the dominant talkers: 42.61 DER on DIHARD III 5+ versus LS-EEND's 42.98. On DIHARD III and CALLHOME 4+, streaming even beats the offline model. The authors blame a train-inference mismatch: offline Sortformer trained on 90-second slices struggles on long recordings, while streaming uses a fixed window.
Cutting latency from 10 s to 0.32 s hurts DER only modestly. On an RTX 6000 Ada with batch 1, RTF is 0.005 at 10 s latency, 0.093 at 1.04 s, and 0.180 at 0.32 s. Quoted latency is input-buffer delay, not compute.
Among published streaming end-to-end numbers at one-second latency, this is one of the strongest, without attractors and without permutation search on the cache. Weights and code sit in NVIDIA NeMo, with a 4-speaker streaming checkpoint on Hugging Face. For live meeting captions, this path is cleaner than chopping an offline Sortformer and realigning chunks.
For anyone already on offline Sortformer, AOSC is the same arrival-order interface made online. Speaker indices that stay stable across chunks are exactly the supervision signal the original Sortformer paper wanted for streaming multi-talker ASR.
The design still caps at four speakers; extending to eight is listed as future work. The 5+ numbers are "track the four dominant talkers," and extra people get dropped. CH109 slightly overlaps CALLHOME part1 used in training. Post-processing parameters are tuned separately on DIHARD III Dev and CALLHOME part1, so they are sensitive to annotation style.
Several baselines omit subsets, so "state of the art" only holds in cells that actually have a number. RTF 0.180 at 0.32 s latency will not meet the quoted delay on a weaker chip or a larger batch. The paper never plugs the streaming diarizer into streaming ASR for word error, which remains a claim.