DEFINE: Exemplar-Guided Accent Control for Zero-Shot TTS
Ambuj Mehrish, Abhinaba Roy, Alex Ivanov, Tawsif Ahmed, Dorien Herremans
cs.SD, eess.AS
2026-09-27
LoRA-adapted F5-TTS takes voice and accent from separate clips. Seen-accent accuracy rises from 6.5% to 19.6%, matching F5-TTS plus Seed-VC with +0.024 speaker similarity.
Zero-shot TTS clones an unseen speaker from a few seconds of audio, and that same clip carries the speaker's accent. Training speech usually shows each person with one native accent, so the loss never has to split the two. Asking for this voice with a Scottish accent has no separate control.
An accent label, or a vector learned from labeled speech, can steer accent directly. A new accent then needs new labels and another adaptation. The other route synthesizes first and converts afterward, as in F5-TTS followed by Seed-VC. The extra waveform pass has to hang on to the speaker, and accent strength is no longer a knob inside the synthesizer.
DEFINE takes two clips. The speaker prompt decides who is talking. A short accent exemplar decides the accent. Inference reads no accent label. After training, one scalar w sets the strength.
The backbone is F5-TTS: non-autoregressive conditional flow matching, a vector field that carries noise onto a target mel spectrogram. The 343M backbone stays frozen. Rank-16 LoRA, small matrices trained beside frozen weights, is applied to the attention query and value projections, the feed-forward layers, and adaLN. Trainable size is 6.3M.
Accent clips pass through frozen XLS-R, a multilingual self-supervised speech model, read at layer 15. A layer-wise accent probe chose that layer. Masked-mean frame features go through an MLP to 128 dimensions and are averaged if several clips are given. Training uses one clip.
The synthesis loss does not teach this global vector. Remove the prototypes and no value of w beats switching the accent off. Training therefore keeps 23 learnable accent prototypes. The encoder is pulled onto its own prototype and pushed away from the others with a contrastive term at temperature 0.1. Half the time the decoder conditions on the encoder output, half the time on the prototype. Stop-gradient keeps the synthesis loss from updating the encoder, so the prototypes become vectors the decoder will follow and the encoder only learns to match them. At inference the table is dropped. Accents outside those 23 are built from audio alone, and the exemplar speakers do not appear in training. A speaker classifier with gradient reversal strips voice identity out of the accent vector.
The accent vector shifts the timestep embedding and the text embedding. Each shift is layer-normalized with no affine transform, then scaled to the root-mean-square of the embedding it modifies. Skip that scale and the added term is about 100 times smaller than the timestep embedding, small enough that training ignores it. The timestep drives adaLN, so every block sees the accent. With probability 0.15 the accent vector is zeroed, which keeps a no-accent branch.
Each solver step mixes three fields: the no-accent prediction, an unconditional prediction with coefficient 2 that drops both text and the speaker prompt, and w times the accent residual. That is classifier-free guidance pointed at accent. At w=0 the accent term is off but LoRA stays on, so the run is not stock F5-TTS. Sequence evaluations go from two to three. The cascade still converts the whole waveform on top.
The test grid has 1,764 items: 1,134 seen, 378 held-out (Australian, Irish, Filipino), and 252 out-of-domain (Welsh, West Indian). Held-out accents are classes in the 15-way probe but were withheld from TTS training. Out-of-domain accents were not in TTS training. ACC is logistic regression on XLS-R layer 15, with chance at 0.067. SPK is the cosine similarity of ECAPA-TDNN embeddings between the output and the speaker prompt. WER is Whisper large-v3. Predicted quality is UTMOS. Exemplar results use w=15, chosen on validation items held out of this grid. Lookup by accent label only covers training accents and is an upper bound.
| Split | System | ACC | SPK | WER | UTMOS |
| Seen | Real speech | 0.436 | n/a | n/a | 3.08 |
| Seen | F5-TTS | 0.079 | 0.732 | 0.071 | 3.99 |
| Seen | Cascade | 0.188 | 0.600 | 0.056 | 3.92 |
| Seen | Accent off | 0.065 | 0.694 | 0.063 | 4.06 |
| Seen | Exemplar | 0.196 | 0.624 | 0.067 | 3.93 |
| Seen | Prototype lookup | 0.354 | 0.562 | 0.089 | 3.77 |
| Held-out | Cascade | 0.175 | 0.603 | 0.076 | 3.89 |
| Held-out | Exemplar | 0.127 | 0.654 | 0.062 | 4.01 |
| Out-of-domain | Cascade | 0.083 | 0.601 | 0.047 | 3.94 |
| Out-of-domain | Exemplar | 0.087 | 0.640 | 0.074 | 3.97 |
On seen accents the exemplar path reaches ACC 0.196 against 0.188 for the cascade, with SPK 0.024 higher, UTMOS 3.93 versus 3.92, and WER 0.067 versus 0.056. Stock F5-TTS scores SPK 0.732 because accent barely moves (ACC 0.079). Out of domain, exemplars score 0.087 and the cascade 0.083; the paper calls that gap statistically indistinguishable. Against accent-off at 0.048 the paired gain is +0.040 (95% interval [+0.004, +0.079]), and against F5-TTS at 0.024 it is +0.064 ([+0.032, +0.099]). Absolute accuracy still sits on the chance line. Held-out accents favor the cascade, 0.175 to 0.127, and w=23 only reaches 0.151. Out-of-domain WER is 0.047 for the cascade and 0.074 for exemplars, so intelligibility does not tie.
SPK falls monotonically with w. Exemplar ACC peaks at w=19 for seen and out-of-domain accents, at 0.214 and 0.111. Lookup keeps rising, to 0.385 at w=23. Reading the accent from the clip saturates first. With the vector placed by the table, injection can still push accent higher.
Ablations on the 378 held-out items keep only the largest w that stays within 0.15 UTMOS and 0.10 SPK of accent-off. Without prototype anchoring that weight is 0 and ACC is 0.079. The contrastive prototype term adds about 0.053. Cross-attention pushed to w=23 reaches ACC 0.175, but UTMOS falls to 1.52 and SPK to 0.22. Three exemplars versus one change seen ACC by 0 (interval ±0.058) and held-out ACC by +0.003, with +0.019 SPK. Nearest-prototype hit rate is 0.873 on training speakers and 0.540 on unseen speakers, against chance near 0.043.
The listening test used 11 people and 110 votes, seen accents only. Every DEFINE sample was prototype lookup at w=11. The exemplar path was not in the test. Votes were 77, 23, and 10 for DEFINE, the cascade, and F5-TTS. The same model at w=0, 11, and 23 drew 11, 52, and 47 votes. Guidance off falls far under the one-in-three chance line. The two active weights are not separable.
What you can ship is one model, one synthesis pass, and no accent label. Relative to the cascade, the whole waveform conversion goes away. On seen accents the probe matches and speaker similarity is higher. Out of domain the scores meet near 0.08. w is applied after training, and the trade between similarity and accent strength is monotonic, so the operating point can be picked later.
Lookup ACC of 0.354 shows the decoder follows an accent vector once that vector is in the right place. The exemplar path at 0.196 is short because the encoder misreads unseen speakers. 6.3M trainable parameters against 343M is a small patch. Capacity is not the ceiling. Held-out accents are still a job for the cascade.
Held-out accents lag, and nearest-prototype accuracy falls from 0.873 on training speakers to 0.540 on unseen ones. Extra exemplar clips do not fill that hole.
The probe shares XLS-R layer 15 with the encoder. Lookup never uses those features at inference and still scores 0.354. The cascade never uses them and still scores 0.188 on seen accents, so a probe biased toward the encoder does not explain the ranking. The probe is coarse on its own: real seen-accent speech scores only 0.436.
Eleven listeners, targets limited to England, India, and the US, and every clip from lookup. At the listening weight w=11, lookup ACC is 0.326. The exemplar path at w=15 is 0.196. That preference does not carry over to the path that runs without labels.
Training includes 94.7k Seed-VC mismatched pairs, accent from Common Voice and timbre from studio speakers, plus 137k real recordings. Held-out and out-of-domain accents were excluded from the converted pool, so the out-of-domain numbers do not rest on that synthetic audio. The seen-accent tie with the cascade does sit next to Seed-VC outputs seen in training. Speaker prompts are studio recordings from a commercial TTS system, and the paper states they yield the highest speaker similarity in the table. Real-speech UTMOS runs from 3.06 to 3.18 while the synthesizers sit near 3.9. The paper does not explain the gap. The score is only useful across systems.