Rethinking Automated Voice Similarity by Shifting from EER to Embedding Geometry
Szu-Chi Chen, Jia-Kai Dong, Yi-Cheng Lin, Sung-Feng Huang, Hung-yi Lee
eess.AS, cs.SD
2026-09-28
EER tracks human voice similarity at Spearman +0.07 across 35 setups; deff tracks it at -0.95. Compressing AM-Softmax embeddings to 3-D raises ρalign from 0.08 to 0.74.
TTS and voice conversion need a cheap score for "does this clip sound like the reference speaker." Listening tests are scarce, so the field extracts speaker embeddings, takes cosine similarity, and selects the extractor with the lowest equal error rate (EER) on speaker verification. The working assumption is that a stronger verifier also tracks graded human timbre similarity.
That assumption was left untested. Verification only asks whether two clips share an identity. Listeners give a 1-6 rating and can tell "very close" from "somewhat close." Researchers at National Taiwan University and NVIDIA train standard speaker embedders on identity labels alone, with no perceptual supervision, and ask two questions: does lower EER mean closer agreement with listeners, and which geometric property of the embedding space actually moves with that agreement.
Perceptual alignment ρalign is the Spearman rank correlation between embedding cosine similarities and mean listener ratings on different-speaker pairs in VoxSim. Thirteen listeners scored pairs from 1,251 VoxCeleb1 speakers; 16,905 of 27,697 unique pairs are different-speaker. Same-speaker pairs are held out because an identity-only baseline that scores 1 for same-speaker and 0 otherwise already reaches SRCC 0.715 on the full set and would swamp differences among embedders.
Effective dimensionality deff counts how many independent directions a model spreads its speakers over. Utterance embeddings are averaged into 1,251 speaker centroids, L2-normalized and centered. deff is the participation ratio of the PCA eigenvalues of that centroid covariance: (sum of eigenvalues) squared, divided by the sum of squared eigenvalues. Large deff means speakers occupy many directions evenly; small deff means a flatter cloud. Classification losses (AM-Softmax, AAM-Softmax) learn a class center per speaker, normalize embeddings, and add a cosine margin, which pushes identities apart across the sphere and tends to inflate deff. Metric losses (Prototypical, Angular Prototypical) pull a held-out utterance toward a prototype built from the speaker's other clips in the batch; they shape relative distances and do not need to fill every direction. Human voice-similarity judgments already fit a low-dimensional space, so this geometry is a plausible match.
Five model conditions, seven losses, three seeds. ECAPA-TDNN and ReDimNet-B2 use 192-D embeddings, Fast ResNet-34 uses 512-D, and two conditions put an ECAPA head on WavLM Base+ (20 epochs with a frozen trunk, then 5 epochs unfrozen). Training data is VoxCeleb2-dev, 5,994 speakers. EER is cosine scoring on full recordings from VoxCeleb1-O. A second grid retrains ECAPA at output widths 512, 32, 10, and 3.
Across 35 training conditions, Spearman(EER, ρalign) is +0.07 (p=0.71). Inside a single architecture the sign flips from -0.68 to +0.61; no permutation test rejects zero (smallest p=0.11). Four public SOTA checkpoints sit in the low-EER, poor-alignment corner of the ECAPA plot.
The training objective is what moves alignment. In every model condition the best and worst losses differ by more than 3× in ρalign, and EER barely moves. Prototypical losses land at 0.395-0.499; classification losses land at 0.080-0.298; AM-Softmax finishes last in all five conditions. On ECAPA, swapping AAM-Softmax for Angular Prototypical raises ρalign from 0.111±0.013 to 0.406±0.007 at a tied-best EER of 1.47%. On WavLM-p2, Angular Prototypical posts the lowest EER in the matrix at 1.09% with ρalign 0.401; AM-Softmax under the same condition is 1.40% EER and 0.102 alignment.
| Setup | EER% ↓ | ρalign ↑ | deff |
| ECAPA + Angular Proto. | 1.47 | 0.406 | 33.0 |
| ECAPA + AAM-Softmax | 1.47 | 0.111 | 74.5 |
| ECAPA + AM-Softmax | 1.62 | 0.080 | 98.3 |
| Fast ResNet-34 + Angular Proto. | 2.35 | 0.499 | 21.2 |
| Fast ResNet-34 + AM-Softmax | 2.19 | 0.158 | 64.0 |
Pooled Spearman(deff, ρalign) is -0.95 (p<0.001). Within conditions it runs from -0.71 to -1.00; four of five permutation tests are significant (p≤0.007). ReDimNet-B2 is the weak cell (p=0.088). Models that spread speakers over fewer directions match listeners better. deff uses only centroids, no listening labels.
Forcing the geometry confirms the direction. On ECAPA, shrinking AM-Softmax from native 192-D to 3-D drops deff from 98.3 to 1.5 and lifts ρalign from 0.080 to 0.738, while EER rises from 1.62% to 20.48%. Angular Prototypical at width 3 reaches ρalign 0.666 and EER 13.60%, a smaller gain, because that loss already used fewer directions. Widening to 512-D changes almost nothing; nominal width is only a ceiling. At width 3, two of three AAM-Softmax seeds collapse (deff≈1.1, EER>40%), the only alignment drop in the grid. Compression buys alignment and spends verification accuracy.
TTS and VC papers treat "the speaker model with the lowest EER" as the automatic timbre metric. This work breaks that default. Same backbone, same data, a loss swap multiplies listener agreement by three to four without giving up EER. Pushing alignment still higher by shrinking the embedding is available, but EER then enters the human verification band: listeners sit near 17.7% EER on VoxSim and 15.8-26.5% on VoxCeleb1; the 3-D AM-Softmax model is at 20.5%.
The practical lever is the loss. For a voice-similarity proxy, prototypical and Angular Prototypical beat AAM-Softmax. deff is a geometric filter that needs no listening labels. The backbone can stay; the objective and the scoring criterion should not.
There is no dedicated limitations section. All listener scores are on real VoxCeleb recordings. None of these embedders is plugged into a TTS or VC pipeline to check whether rankings of synthetic speech flip the same way, which is the application argued in the introduction. VoxSim has 13 listeners on English celebrity speech; whether the -0.95 correlation survives other languages and messier recording conditions is untested.
ρalign drops same-speaker pairs, so it measures how alike two different people sound, not how alike a clone is to its source. If a downstream task mainly cares about matching the reference identity, the metric is off-axis. The 0.74 alignment at width 3 comes with about 20% EER and unstable training, so that model is not a verifier. Public checkpoints appear as four stars on the ECAPA figure only; there is no census of speaker models used in production.