How streaming speech-to-speech models decide when to speak: READ/WRITE gate architecture

Gauri_the_great · x · 2026-11-03

A detailed breakdown of how streaming speech-to-speech translation models decide when to start speaking before the speaker finishes:

Pipeline: encode partial speech → predict commitment → generate target text → synthesize speech.
Audio encoding: 640ms source blocks with 160ms lookahead; a block-causal adaptor compresses 16 encoder states into 8 positions, blocking attention into the next block.
Thinker: processes interleaved source-audio features and committed target text; an MLP maps the final hidden state to p(WRITE) — below threshold τ it READs another block, above it WRITEs a translation span.
Talker: receives a channel-wise weighted mixture of 8 Thinker layers (not just the final text state), generates codec tokens converted to audio via a frozen flow-matching decoder and vocoder.
Timing supervision: semantically aligned bilingual spans are force-aligned to audio; source endpoints become WRITE targets, intermediate boundaries READ targets. Training combines text, codec, and gate classification losses.
Practical control: raise τ while audio remains buffered to gather more context; lower it when the buffer empties for faster response.
Caveat: the gate predicts annotated commitment decisions, not translation correctness.

More from Research