EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation
Kuan-Po Huang, Haohe Liu, Puyuan Peng, Haibin Wu, Zhaoheng Ni, Hung-yi Lee, Jinwon Lee, Neha Chachra
cs.SD, cs.CL, eess.AS
2026-09-30
EmoRES boosts residual emotion steering 3x without training. IEMOCAP vs CoCoEmo: ρ +26.13 points (118.8% relative); listener dominant-emotion hit rate up to +35% relative.
Emotion-conditioned TTS already takes labels, natural-language instructions, or emotion embeddings. Those native controls often still leave the utterance close to neutral. Agents, audiobooks, and dubbing need the requested emotion without wrecking intelligibility, speaker identity, or naturalness.
Retraining on more labeled emotional speech is expensive and still struggles with mixtures. Vector steering freezes the backbone and adds a direction, usually the mean hidden state of emotional speech minus the mean of matched neutral speech. CoCoEmo made this work on LM-based TTS, but it treats each emotion vector as one stick scaled by a single strength α. The shift that leaves neutral and the contrast that picks a category are locked to the same ratio. Asking for more sad also scales up general affect, so the two jobs share one knob.
The emotion vector ve is the mean activation of emotion e minus the neutral mean. Around the centroid of all non-neutral emotion means, it splits in two.
The shared piece is "turn on affect." The residual is "which one." CoCoEmo turns both with one knob.
EmoRES gives each piece its own coefficient: vRES = λc × shared + λr × residual. When both equal 1, the centroid cancels and the rule is exactly CoCoEmo. For mixtures, the shared coefficient stays fixed and the residual is a proportion-weighted blend. After the add, the hidden-state norm is restored so only direction changes, which keeps large α from inflating magnitude.
Tables use λc = 1, λr = 3. Every method is rescaled to the mean Euclidean length of the original ve at that layer, so the comparison is direction, not who pushes harder. The intervention sits at the attention output, on CoCoEmo's most emotion-separable layers: 1, 6, and 8 of IndexTTS-2's semantic LM, and 14 and 17 of CosyVoice2's text-to-token LM. Vectors come from paired neutral-emotional utterances in ESD, CREMA-D, and RAVDESS (same speaker, same text), gated by a speech-emotion recognizer and a quality filter. Weights stay frozen. Native conditioning stays on.
Hyperparameters are tuned only on CREMA-D. IEMOCAP contributes no speakers and no recordings to vector construction. Rows below use α = 5, λc = 1.
| Backbone | Metric | No-steer | CoCoEmo | EmoRES (λr=3) |
| IndexTTS-2 | ρ | n/a | 22.00 | 48.13 |
| IndexTTS-2 | H-Rate | n/a | 64.34 | 77.29 |
| IndexTTS-2 | TEP | 30.76 | 36.22 | 41.11 |
| CosyVoice2 | ρ | n/a | 39.13 | 52.10 |
| CosyVoice2 | H-Rate | n/a | 70.90 | 77.82 |
On IEMOCAP, ρ rises 26.13 and 12.97 points (118.8% and 33.1% relative). H-Rate rises 12.95 and 6.92 points (20.1% and 9.8% relative). E-SIM on IndexTTS-2 goes from 65.32 to 67.67. Speaker similarity beats CoCoEmo in every comparison; WER is lower in most. Native conditioning alone is weaker on TEP and E-SIM. CREMA-D shows the same pattern: ρ 25.24 → 45.87 on IndexTTS-2, 32.48 → 51.94 on CosyVoice2.
Human labeling on IEMOCAP, at least nine annotators per clip:
| Backbone | Dom-hit | Fidelity | Tie-adjusted naturalness pref. |
| IndexTTS-2 | 54.72% → 73.89% | 60.05% → 66.32% | 63.80% |
| CosyVoice2 | 58.18% → 77.88% | 50.80% → 59.60% | 60.32% |
Dom-hit relative gain peaks at 35.0%, fidelity at 17.3%. Without steering, IndexTTS-2 Dom-hit is 17.78%. On IndexTTS-2 pairwise naturalness, 50.83% pick EmoRES, 25.93% call it a tie, 23.24% pick CoCoEmo.
Ablations keep α and vector norm fixed and only change λc and λr. Shared-only drops ρ to 5.30% and 7.20%, and cuts IndexTTS-2's neutral SER posterior from 36.51% to 7.49%: it leaves neutral without naming the category. Residual-only lifts ρ and H-Rate but loses TEP and E-SIM versus the combined direction. Equal weights are still not the best row; λr = 3 is. Past 3, gains shrink. In emotion2vec space on 486 single-emotion CREMA-D clips from IndexTTS-2, no-steer probe accuracy is 56.9% with silhouette -0.035; CoCoEmo 82.5% / 0.335; EmoRES 86.9% / 0.460.
This is an inference-time knob for frozen LM-based TTS: no extra emotion labels, no learned subspace, no weight updates. If CoCoEmo already runs in the stack, setting residual weight to about 3 and keeping the shared shift at 1 moves mixed-emotion rank metrics and listener identification, and pairwise naturalness does not fall.
It is geometric reweighting, not a new model. The useful setting is narrow: frozen backbone, weak native emotion control, mixture proportions. For anyone extracting steering vectors, the split is worth more than the number 3. Shared means some emotion. Residual means which one.
The paper's own follow-ups: token- or segment-level control for emotion that changes inside an utterance, and tests on more emotions, languages, corpora, and TTS architectures.
Only four categories and two English backbones. λr = 3 and the layer set were chosen on CREMA-D, which also sits in the extraction library, so the development numbers can leak. IEMOCAP is the clean holdout. Objective scores depend on SER posteriors and emotion2vec, and extraction is SER-gated, so the same family of models both builds the directions and grades them. Human scores offset that in part; CosyVoice2 fidelity is still 59.60% absolute. Random-direction controls live in an appendix with no numbers in the main text. Native conditioning stays on, so the comparison is conditioning plus steering. There is no cost or ceiling comparison against retraining. Appendices mention single-emotion requests, steering with native conditioning off, and other SERs, without numbers in the body.