StyleStream 2.0: Fast and Controllable Streaming Voice Style Conversion
Yisi Liu, Nicholas Lee, Gopala K. Anumanchipalli
NeurIPS 2026
cs.SD, cs.AI
2026-10-21
UC Berkeley's StyleStream 2.0 cuts streaming voice conversion latency to 520 ms with a 4-step MeanFlow stylizer, and adds text-based voice design and editing.
Voice style conversion (VSC) rewrites how an utterance is spoken: timbre, accent and emotion switch to a target voice, the words stay put. StyleStream 1.0, the predecessor from the same Berkeley group, was the first zero-shot VSC system that runs streaming, and it shipped with two problems. End-to-end latency sat around one second on consumer hardware, too slow for calls or live interpretation. And a target voice could only be specified by supplying a reference clip; if you wanted 'a middle-aged man, Indian accent, angry,' there was no way to just say so.
The second problem is the structural one. Prompt-controlled TTS has made natural-language voice specification routine, but TTS synthesizes from text and does not convert speech you already have. Text-guided speech-to-speech editing exists, typically along a single axis and offline. No prior system combined streaming, holistic style transfer, and text-based design plus editing.
The two-stage skeleton carries over. A Destylizer extracts content features stripped of style (frozen HuBERT-Large plus six conformer blocks, trained with an FSQ bottleneck and an ASR loss so the representation keeps little beyond linguistics; both are discarded at inference). A Stylizer rebuilds the mel-spectrogram from those features plus a target style, and a causal Vocos vocoder produces the waveform. The Destylizer is reused untouched; every change lands in the Stylizer.
Latency work comes in two moves. The multi-step diffusion Stylizer is replaced by a pixel MeanFlow (pMF) model: the network predicts the clean mel directly, supervision happens in MeanFlow's average-velocity space, and classifier-free guidance is distilled into the network with the guidance scale as an input. That buys one forward pass per sampling step and cuts steps from 16 to 4. The second move matters more, because fast sampling alone does not survive chunk-by-chunk inference; each chunk sees little context and boundaries degrade. Borrowing Self Forcing from autoregressive video diffusion, the Stylizer is fine-tuned on rollouts of its own output: generated history fills a past-output channel, the loss covers only the current chunk, and gradients stop at the history. Training then matches what streaming inference actually feeds the model.
Control is where the redesign is cleanest. Version 1.0 conditioned the Stylizer on both an unmasked reference mel context and a style embedding, leaving style spread across two channels that text cannot touch. Version 2.0 drops the mel context, so all style lives in one compact embedding, and design or editing reduces to predicting or modifying that embedding. A Voice Controller, a text-conditioned DiT in embedding space modeled on FLUX Kontext from image generation, maps a prompt to an embedding for voice design and edits a source embedding under an instruction for voice editing; text is encoded by a frozen Flan-T5. Controllable attributes are age, gender, accent and emotion. Since the embedding now carries everything, the old TDNN style encoder gives way to a perceiver resampler over WavLM's twelve layers; under the same objective, the old encoder drops S-SIM from 0.879 to 0.823.
Standard VSC is evaluated on 3000 source-target pairs covering five emotions and five accents:
| System | Setting | Steps | WER ↓ | S-SIM ↑ | A-SIM ↑ |
| StyleStream 1.0 | offline | 16 | 7.47 | 0.869 | 0.699 |
| StyleStream 2.0 | offline | 4 | 7.00 | 0.879 | 0.705 |
| StyleStream 1.0 | streaming | 12 | 11.41 | 0.842 | 0.656 |
| StyleStream 2.0 | streaming | 4 | 12.02 | 0.866 | 0.684 |
| Vevo 2.0 (ASR+TTS cascade) | offline | 32 | 5.79 | 0.843 | 0.676 |
Offline, StyleStream 2.0 posts the best style similarity across the board, with S-SMOS 4.32 and E-SMOS 4.35 also first. It loses WER to Vevo 2.0, which is effectively an ASR plus voice-cloning TTS cascade and holds a natural intelligibility edge. Streaming runs at 520.7 ms end-to-end on an RTX 4060 (300 ms chunk, 80 ms lookahead, 140.7 ms per-chunk processing), twice as fast as 1.0; streaming style similarity beats 1.0 everywhere except a marginal E-SIM gap (0.765 vs 0.771), while WER ticks up from 11.41 to 12.02. Across the 160-600 ms chunk sweep, 2.0 stays real-time throughout, while 1.0 only qualifies at 600 ms with 598.4 ms of processing.
No native baseline exists for text control, so the authors assembled cascades. For voice design, the offline model hits 70.67% age accuracy against 41.11% for CapTTS and 44.33% for Qwen3-TTS, 99.44% on gender, 52.73% on accent (level with CapTTS), and 32.20% on emotion, behind Qwen3-TTS's 36.33%. For editing, offline accuracy on the edited axis is highest across the board (age 71.67%, gender 99.17%, accent 54.33%); Vevo manages 16.89% on accent edits. The most informative ablation: removing autoregressive fine-tuning pushes streaming WER from 12.03 to 19.54 with style similarity essentially flat, so the Self Forcing stage fixes chunk-boundary coherence and intelligibility, not style.
For anyone building real-time voice features, the headline is usable now: 520 ms end-to-end, real-time on a consumer GPU, no ASR-plus-TTS cascade in the path, and a text interface that removes the reference-audio hunt. Two ideas transfer beyond voice. Few-step generation plus autoregressive rollout fine-tuning separates 'sample fast' from 'stream stably' as two solvable problems. And compressing all controllable information into a single latent, then training one unified text-conditioned model there, is FLUX Kontext's recipe validated on speech. The honest framing: the core conversion loop is an incremental upgrade over 1.0, Destylizer untouched; the genuinely new capability is text-based design and editing.
Emotion control is weak in absolute terms: 32.20% design accuracy, 14.67% in editing, the worst of the four attributes. Preservation of non-edited attributes in editing trails the baselines (speaker preservation 0.7899 vs Vevo's 0.9092); the authors argue those baselines preserve because they barely edit, which is half fair, but subjective SP-MOS only ties (3.58 vs 3.64). Streaming WER of 12.02% sits well above the 7.00% offline figure, so streaming still costs intelligibility. Training used roughly 50k hours of English from Emilia, so 'accent' means accents within English and nothing cross-lingual was tested. Every text-control comparison runs against cascades the authors built themselves, since no native competitor exists. The paper also says nothing about safeguards against voice-spoofing misuse.