Fish Audio Drama 3 Technical Report
Dawei Han, Shidong Li, Shijia Liao, Jiahua Liu, Songting Liu, Qingzheng Wang, Yuxuan Wang, Xin Chen, Yifan Cheng, Tianyu Li, Gangin Park, Xuehao Yang, Chang Zeng, Haoyang Zhang, Ruoyi Zhang, Xiangyu Zhang
cs.SD, cs.CL
2026-10-13
Open Drama 3 (35B MoE, 3B active) turns text into speech you can revise and edit, decoding at 271 fps on an H200 versus S2's 177, with the lowest cloning error in five languages.
Large text-to-speech systems already read cleanly, and speaker similarity is close to a real voice. Audiobooks, audio drama, and film or game dubbing ask for a performance. A line needs a written direction, another pass after the director hears it, a place among other characters, and a one-word fix that leaves the rest of the take alone.
Fish Audio S2 already had inline instructions and native multi-speaker reads. Four limits stayed. Full attention swallows every reference and every earlier take, and a long session runs to thousands of audio tokens. Control fires once: a finished take cannot take a note and be redone. Changing one word regenerates the whole take. Rewards built on word error and speaker similarity punish laughs, long pauses, and whispers, the deliveries that move the transcript and the timbre those scores expect.
Drama 3 keeps S2's discrete tokenizer, based on the Descript Audio Codec. Audio at 44.1 kHz becomes about 21.5 frames per second with ten codebooks. Codebook 0 is semantic. The next nine are residual vector quantizers, each one finer. Encoding and decoding are causal, so the codec streams. One second of speech is about 21.5 positions in the backbone. There is no audio encoder. References enter only as these tokens.
The backbone starts from the language model of Qwen3.5-35B-A3B and has 40 layers. Three of every four are Gated DeltaNet, gated linear attention that stores history in a fixed-size state. The fourth is gated full attention, there to retrieve a reference timbre or a line far back in the script. Each layer is a mixture of experts: 8 of 256 routed experts plus a shared expert, about 3B of 35B parameters active per token.
S2 summed all ten codebook embeddings into the backbone. Drama 3 feeds back only codebooks 0 and 1. Finer codes skip the backbone, so the model's own sampled detail is less able to drag later frames off, while a coarse acoustic view remains. Acoustic codes are predicted by a codebook multi-token-prediction decoder: 13 full-width layers with causal attention over the whole sequence, reference detail included. Four shared layers run first. Each of the next nine writes one acoustic codebook and reads the code just chosen. Training is teacher-forced and parallel across time. S2's Fast AR is a small network inside one frame, conditioned on a single projected backbone state.
Every task is a chat. Loss is only on the assistant audio. Pre-training runs on the order of hundreds of billions of tokens and about 80 languages, with the Muon optimizer. Context goes from 8K to 16K. Long recordings, streaming text, and infilling arrive in the third stage. Instruction tuning extends to 32K and teaches point tags, tags around a few words, a style for the whole line, and free-form notes. A later turn can ask for something softer or warmer, with the previous take left in context. A full scene is provided at once, so one line can depend on the lines after it.
Reinforcement learning drops word-error and similarity rewards. A smaller speech language model, trained on human preferences, is the only score. It reads audio tokens directly, references and script included, with no speech recognizer in the reward path. Scores are clipped and turned into a win rate within the group. Each prompt draws 8 takes. Updates use Group Relative Policy Optimization (GRPO) with no KL penalty. The decoder stays frozen, so acoustic codes get no preference gradient. Each step uses 1,536 prompts times 8 takes and the schedule is short, which in development was steadier than many steps on small batches.
Serving is bound by memory bandwidth. At batch size 1, S2 reads about 15.6 GB of weights per frame and has about 4.6B parameters. Drama 3 reads about 6.0 GB: routed experts are block-scaled FP8, and the decoder runs once per frame. On one H200 a single stream reaches 271 frames per second, against 177 for S2 under the same measurement.
Every public benchmark uses the same checkpoint, sampled at temperature 0.7 with top-p 0.7 and top-k 30. Silent, too-short, or runaway outputs are regenerated before scoring.
On Expresso, styled human readings of a sentence are less similar to a neutral reference than the default reading. Whispering drops similarity by 0.310, clearly enunciated speech by 0.019. A similarity reward therefore picks the default in 62.0% to 99.7% of pairs. A laugh raises word error by 0.79 points. When the style is written into the instruction, the preference model picks the whisper in 62.2% of pairs and the laugh in 64.4%. Without the instruction those shares are 10.6% and 27.1%.
| Setup | Metric | SFT | After RL |
| Seed-TTS Chinese | First-take CER | 1.66 | 1.00 |
| Seed-TTS Chinese | CER after regen | 0.84 | 0.83 |
| Seed-TTS English | First-take WER | 3.49 | 2.20 |
| Seed-TTS English | WER after regen | 1.60 | 1.54 |
| Failed signal check | zh / en | 2.82% / 3.40% | 1.24% / 1.84% |
There is no transcription reward. First-take failures still fall, from 2.82% to 1.24% in Chinese and from 3.40% to 1.84% in English. On takes that pass the check, Chinese CER moves from 0.84 to 0.83 and English WER from 1.60 to 1.54.
On LibriSpeech-PC, one reference gives WER 1.85, similarity 0.605, and UTMOS 4.31. Three references move similarity to 0.633 and leave error and UTMOS unchanged. OmniVoice reports 1.30 and 0.729, Qwen3-TTS 1.60 and 0.704, CosyVoice 3 1.59 and 0.694. Error and similarity both trail. UTMOS at 4.31 is above the 4.28 of OmniVoice and CosyVoice 3 and below Qwen3-TTS at 4.41.
On the 24-language MiniMax set, Drama 3 has the lowest error in the table for Chinese, Dutch, English, German, and Turkish. Chinese CER is 0.63, against 0.66 for ElevenLabs v4 and 0.73 for S2. English WER is 1.13, and every other system in the table is at or above 1.56. It is lower than OmniVoice in 15 languages. Cantonese CER is 22.70, worse than OmniVoice at 17.71. Japanese is 5.02, worse than S2 at 2.76.
Streaming barely moves English Seed-TTS. On all 1,088 sentences, streamed WER is 1.86 and non-streamed WER is 1.83. Similarity is 0.640 in both.
On Ming-Freeform Chinese, insertion and substitution similarity are 0.909 and 0.898, against 0.89 for CosyEdit2, but insertion error is 2.15 against 1.38. Deletion error is 7.82 against 8.23. English insertion error is 3.03, worse than SSR-Speech at 1.75. Against Cartesia Sonic-3 infill, average Chinese error is 3.69 versus 8.64 on the basic split and 2.35 versus 6.53 on the fuller split.
On Step-Audio-Edit, Gemini-2.5-Pro labels each take. Emotion accuracy rises from 55.5 to 58.2 over three rounds, style from 76.5 to 81.8. In the Prompt-Fixed setting, Step-Audio-EditX goes from 53.6 to 69.3 on emotion and from 45.6 to 63.8 on style. Paralinguistic events are inserted with a point tag and an infill, lifting Drama 3 from 1.32 to 2.32, against 1.91 to 2.89 for EditX.
One open checkpoint clones, takes direction, revises, plays several characters, edits a span, and speaks text as it arrives. By total parameters the report calls it the largest open speech-generation model so far. A token activates about 3B, and a frame reads less than two fifths of the weights S2 reads, which is why one stream is faster.
Expresso puts a number on the conflict between expression and the usual rewards. After preference RL, first-take junk drops and intelligibility on clean takes holds. Emotion revisions gain only 2.7 points. What can enter a production loop is a note against the previous take, and a repair of one word. English cloning similarity still trails specialist systems.
The report has no limitations section.
Similarity on LibriSpeech-PC is 0.605 with one reference and 0.633 with three, behind OmniVoice at 0.729. The three-reference condition was added for this report. Japanese and Korean error rates are worse than S2. English infill loses on error rate to dedicated editors on several splits. Three emotion rounds stop at 58.2, while EditX reaches 69.3. Style favors Drama 3, but the judge is this report's own Gemini-2.5-Pro run and the EditX numbers are quoted from the earlier paper. On clearly enunciated speech the reward model prefers the styled reading in only 8.1% of pairs even when the direction is written in. Expresso scores human recordings, not whispers produced by Drama 3. There is no listening test. Whole-scene multi-character dubbing has no public number, and neither codebook-MTP nor the two-codebook feedback has a quality ablation. The decoder is frozen during RL. With no KL penalty, the report never measures how far the policy moved or whether the reward was gamed. Some baselines come from papers with their own scoring, and most tables are computed after failed takes were regenerated.