SteerDuplex: Steerable Duplex Speech Dialogue Models
Utkarsh Tyagi, Ramaneswaran Selvakumar, Advait Gosai, Sonal Kumar, Nikhil Barhate, Isabell Sagar, Steven Li, Miheer Bavare, Daniel Quigley, Fabiola Tapia Carrillo, Jose M Patron E, Diego Macías Gutiérrez, Paul Song, Ramani Duraiswami, Dinesh Manocha, Yunzhong He
cs.AI, cs.CL
2026-09-11
SFT on Moshi lifts SteerBench audio-steering all-pass from 20.55% to 65.10%. Two-stage RL raises interruption response from 72.5% to 82.5% but still hacks timing.
Full-duplex speech models listen and talk at the same time. Turn taking, interruptions, and backchannels already work in systems such as Moshi, which lines up two audio streams with a text channel on one clock. Steerability is the missing piece: if a user asks for sarcasm, a retired-professor persona, a slower rate, or a different accent, the model often keeps talking in its default voice.
Scale AI and the University of Maryland measured that gap on matched items with one Gemini 3.6 Flash judge. Moshi's audio-steering all-pass rate is 20.55%. PersonaPlex, which already conditions on voice and role, scores 16.44%. Fluent floor management does not imply control over delivery.
SteerDuplex post-trains the public Moshi 7B backbone. A natural-language system prompt prefixes the assistant text stream. Both audio streams stay silent during the prefix, and prompt tokens are masked from the loss.
Supervised fine-tuning mixes natural conversations with targeted speech and text for instruction following, requested delivery, duplex interaction, safety, and reasoning. The mix has 504,416 audio records and 65,675 text records, sampled at 88% / 12% of training mass, totaling 8,510.9 hours of audio records. That hour count includes repeated source material. Targeted slices include 25,000 steering records, 20,000 multi-turn instruction records, 1,200 duplex records, and 1,300 six-turn math conversations.
SFT is meant to install content and delivery. Timing is left to two-stage RL. GDPO standardizes each reward inside a group of rollouts that share one context, then sums weighted components so a loud scalar cannot dominate. Policy gradients flow through text-stream actions, including padding tokens that mark silent frames; audio codebook actions get no direct policy loss.
Interruption credit requires assistant speech before the interruption, so silence cannot earn a yield score. Reference-audio rubrics grade steerability but do not enter the reported RL objective.
SteerBench holds 390 spoken prompts and 1,067 human-authored binary rubrics (438 audio, 629 text) across tone, persona, style/accent, and speed/length. Audio rubrics compare against fixed reference clips. Aggregation splits three ways: audio-steering all-pass, sample all-pass (text plus audio), and per-rubric pass rate. Passing three of four rubrics is a 75% rubric rate and still a failed example.
Most of the steering gain comes from SFT.
| Model | SteerBench audio-steering APR | Audio MultiChallenge APR | FDB-v2 task mean | VoiceBench overall |
| Moshi | 20.55% | 3.98% | 2.59 | 38.55 |
| PersonaPlex | 16.44% | 6.64% | 2.65 | 30.51 |
| SteerDuplex-SFT | 65.10±1.13% | 13.64±0.28% | 4.17 | 40.87±0.27 |
Audio-steering all-pass rises 44.5 points over Moshi. Audio MultiChallenge task all-pass rises 7 points over PersonaPlex. Sample all-pass still sits between 32% and 51.11%, worst on speed/length and best on persona. Individual rubric pass rates of 63.10%–77.03% do not mean the task finished. Spoken revisions almost never succeed: Audio MultiChallenge voice-editing all-pass is 2.56%, against 0 for both open baselines.
RL mainly moves timing:
| Metric | SFT | +RL |
| Source-clean interruption response | 72.5% | 82.5% |
| Continuation after user backchannel | 71.4% | 80.6% |
| Synthetic pause barge-in (lower is better) | 26.5% | 9% |
| SteerBench rubric pass | 63.75% | 65.22% |
| AudioMC APR | 13.64% | 14.38% |
| VoiceBench | 40.87 | 41.38 |
| FDB-v2 task mean | 4.17 | 4.17 |
FDB-v2 daily tasks drop from 3.74 to 3.41, while safety rises from 4.65 to 4.81. Semantic score on synthetic interruptions slips from 3.94 to 3.88, and mean takeover latency grows by 40 ms. Floor takeovers during user backchannels fall from 9.1% to 4.8%, yet daily-task utterances that stop while the examiner is still speaking rise from 23.6% to 32.3%. Better timing on one event is not better interaction everywhere.
Isolated rewards invent empty speech. On 96 held-out rollouts, a text-rubric-only objective leaves 47 empty outputs. Promptness-only leaves 25 empty and scores zero on the shared duplex diagnostic. A joint composite still leaves 4 empty. Later in stage 2, interruption reward on natural development conversations climbs from 0.450 to 0.793, while continuation after a user backchannel falls from 3.20 s to 2.00 s, below SFT's 2.60 s.
The useful split is content, delivery, and floor control. PersonaPlex already exposes voice and role knobs; it still fails SteerBench audio steering more than four times out of five. A control interface is not the same as following a spoken request to change how you sound.
SFT carries the steerability lift. RL is a timing patch, and the paper documents how timing metrics can be gamed with short or missing answers. Interruption success has to be read next to whether the answer actually finished.
Expectations for deployment should stay modest. The backbone is still Moshi 7B. VoiceBench overall is 40.87 against 63.01 for Gemini Live and 78.42 for GPT-4o Realtime. Audio MultiChallenge all-pass is 13.64%. These numbers measure post-training inside one backbone, not the ceiling of steerable speech models.
SteerBench is controlled English with fixed reference clips. It does not cover other languages, dialects, or recording conditions. Some fine-grained styles remain below the stronger open baseline. Repeated evaluations average the three highest-scoring runs from pools of unequal size, which can inflate scores; FDB-v1.5 uses one decoding pass. CANDOR overlap in supervised data makes official FDB-v1 turn and pause scores diagnostic only.
The reported RL checkpoint yields too readily in some FDB-v2 contexts. The single-stage control has fewer updates, so staging itself is not isolated. Backbone, data, and reward design are not ablated. Moshi's 7B capacity, codec, and streaming stack bound what post-training can do.
Judges are a soft spot. Reference-audio grading can favor one realization of a style even after human validation. Human agreement is about 76% with a reference clip and 62% without. A second-judge study on a shared generation pool finds an RL turn-taking loss (Gemini −0.074, gpt-5.4-mini −0.059), while the broader table shows a small gain. SteerBench and AudioMC have no second judge. The composite reward still leaves 4 empty outputs in 96 probe rollouts; those probes are separate from the reported RL run, and they point the same way: timing rewards lie when read alone.