TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
Freeman Jiang, Ramon Sanabria, Soham Deshmukh, Bandhav Veluri, Simon Michael Vuch Williams, Elliott K. Suen, Garreth Lee, Kevin Yoonho Choi, Takuya Umeki, Riku Kubo, Sathvik Udupa, Chien-yu Huang, Shih-Yun Shan Kuan, Zhuoyan Tao, Satyapriya Krishna, Sefik Emre Eskimez, Yu Tsao, Hung-yi Lee, Shinji Watanabe
eess.AS, cs.CL
2026-08-26
TurnBench labels 30 hours of speech across six conversation types. Among 14 systems, VAP leads with 0.845 EOT recall at 368 ms median; none match humans starting 151 ms early.
A spoken dialogue system has tens to hundreds of milliseconds to decide whether to take, hold, or yield the floor. Evaluation has not kept up. Most corpora cover a single conversation type, and annotation conventions disagree: the Full-Duplex-Bench series uses broad definitions of backchannels, fillers, and interruptions; Talking Turns is tighter but only covers telephone speech. Full-duplex models do not emit turn-taking labels at all, so decisions have to be read back from their audio. Reported gains are then hard to separate from changes in the labeling scheme.
TurnBench treats conversation type as an experimental variable. It scores heterogeneous systems on end-of-turn and interruption detection against one linguistically grounded annotation and one protocol.
The corpus is 30 hours of dual-channel dyadic speech: 154 dialogues, 106 voice actors in 53 pairs, recorded in isolated booths at 48 kHz. Six types are roughly balanced (13–21% each): Casual, Task-Oriented, Instructional, Collaborative, Argumentative, Narrative. A director sets the type and a starting topic and never mentions interruptions, backchannels, or dominance. A 104-hour training set uses the same protocol and is speaker-disjoint from the benchmark.
Annotators mark 17 fine-grained event types. Evaluation collapses them into 7 canonical classes, with Turn, Interruption, and Backchannel as the ones that matter for scoring. Three annotators label independently; Fleiss κ is 0.78 on the canonical map. A gold span needs two agreeing labels and endpoints within ±200 ms; endpoints are medians. 85.8% of annotator events survive. That gold yields 8,197 end-of-turn anchors and 1,151 floor-taking interruption anchors.
A submission is per-conversation, per-speaker event timestamps. The scorer looks in \([t-0.25\,\mathrm{s},\,t+3.0\,\mathrm{s}]\) for the earliest unclaimed prediction, then reports recall, false-positive rate, and signed latency. The leaderboard ranks test recall, and any run above 0.15 FPR is placed below every qualifier. Operating points are chosen on dev under a 0.1 FPR budget.
Fourteen systems are scored: energy VAD, OpenAI Realtime Server/Semantic VAD, Kyutai SVAD, SmartTurn v3, Voice Activity Projection (VAP; predicts future voice activity per speaker), a Mimi endpointer, ESPnet turn prediction, WavLM variants, plus Gemini 3.1 Live and Moshi. Full-duplex models are scored on end-of-turn only, by seating the model on one side of the conversation and reading speech onsets; latency includes generation and, for Gemini, the network.
In smooth floor transfers, humans start speaking a median 151 ms before the current turn ends (281 ms if interruptions are included). No in-budget system matches that earliness without blowing the false-positive cap.
| System | EOT recall / FPR | EOT median latency | INT recall / FPR | INT median latency |
| RMS VAD | 0.718 / 0.632 | −117 ms | 0.996 / 0.445 | 123 ms |
| OpenAI Server VAD | 0.955 / 0.525 | 282 ms | 0.990 / 0.458 | 184 ms |
| OpenAI Semantic VAD | 0.303 / 0.018 | 793 ms | 0.484 / 0.271 | 196 ms |
| VAP | 0.845 / 0.055 | 368 ms | 0.945 / 0.107 | 994 ms |
| Kyutai SVAD | 0.773 / 0.059 | 1007 ms | 0.898 / 0.081 | 559 ms |
| Gemini 3.1 Live | 0.657 / 0.022 | 1234 ms | n/a | n/a |
| Moshi | 0.233 / 0.044 | 702 ms | n/a | n/a |
VAP is the strongest in-budget operating point. End-of-turn recall is fairly stable across the six types. Interruption false positives are not: every model posts a higher INT FPR on Casual than on Argumentative, where backchannels are densest. Acoustic-only gates (RMS, OpenAI Server VAD) saturate recall far above budget. Linguistic endpointers stay in budget and then have to pick speed or selectivity. SmartTurn v3 commits interruptions at 159 ms median with 0.107 recall; VAP waits 994 ms and reaches 0.945 recall.
Voice-agent and full-duplex teams finally have one table that holds rule VADs, endpointers, projection models, and closed Live APIs. The finding is concrete: silence-based VAD treats backchannels as barge-ins; telling a real interruption from a backchannel currently costs about a second. The human 151 ms early start is unmatched under the false-positive constraint.
Conversation type is the actual experimental lever. An endpointer tuned on telephone or meeting speech will light up false interruptions in casual talk. The leaderboard, dev scoring, and an audio viewer are public; test labels for 116 dialogues are held out.
All dialogues are English, studio-recorded, and dyadic. Majority vote drops disagreement that is itself a signal, especially when overlap could be a turn, an interruption, or a backchannel. There is no interruption protocol for full-duplex models, and Gemini latency includes the network. The license is non-commercial and bans voice cloning. Speakers are voice actors, pre-acquainted where possible; types are assigned before recording, so transfer to a real meeting or call center is untested. OpenAI and Gemini numbers are a June 2026 snapshot.