Multimodal Speaker Verification as a Threat to Speaker Anonymization
Ashi Garg, Cristina Aggazzotti, Leibny Paola García-Perera, Nicholas Andrews
eess.AS
2026-07-22
Johns Hopkins shows that aggregating audio, text, and prosody across multiple anonymized utterances breaks voice anonymization: 5 utterances of audio+text cut EER over 15% versus audio-only.
Voice anonymization aims to do one thing: keep what was said, hide who said it. Voice conversion swaps your acoustic voice for someone else's while leaving the words intact, and the VoicePrivacy Challenge standardized the idea. Real-time systems built on it are in production.
The catch is that every existing method and every evaluation looks at a single, isolated utterance, and only at the acoustic layer. Real speech does not come in single utterances. It comes as a conversation, a podcast episode, a chain of calls. The more of it accumulates, the more it leaks: not just the voiceprint, but prosodic habits (pitch, rate) and linguistic choices (vocabulary, syntax). An attacker with several of your anonymized recordings could assemble identity cues that no single clip reveals.
The deeper problem is that the attack models used to grade how safe an anonymizer is also consume one utterance and only the audio. The Johns Hopkins team asks the obvious next question: if the attacker has multiple utterances and uses text and prosody on top of audio, does anonymization still hold?
The paper is built around three questions: does aggregating audio across utterances help (RQ1), do multimodal systems beat unimodal ones (RQ2), and is utterance-level or frame-level aggregation stronger (RQ3). Aggregation happens at two granularities.
At the utterance level, each utterance is encoded independently into an embedding and the embeddings are then aggregated across utterances. Audio uses a learnable query vector that attends over the utterance embeddings with multi-head attention, weighting utterances by how much speaker information they carry. Fusion pairs a WavLM-ECAPA-TDNN audio embedding (a mainstream speaker backbone) with a LUAR text embedding, an authorship-attribution model that pulls identity out of linguistic style. The two are projected into a shared 256-dim space and concatenated into a 192-dim speaker vector. A prosodic branch adds three features: mean F0, speaking rate, and voiced ratio.
At the frame level, frame representations from multiple utterances are concatenated along the time axis before pooling. The reasoning: residual speaker information after anonymization is spread across the temporal axis, so it is better captured before utterance-level pooling than after. Variants include frame-level audio+text and audio+prosody fusion, plus token-to-frame cross-attention (text tokens as queries, acoustic frames as keys and values) to capture pronunciation and articulation. RJCA (Recursive Joint Cross-Attention) uses BiLSTMs and recursive cross-attention to model both intra-modal and cross-modal relations, and a WavLM-Whisper cross-attention variant fuses the two frame-level front-ends.
Attackers come in two strengths: lazy-informed, trained only on original speech, and semi-informed, with access to anonymized training data. The data is the Fisher English telephone-conversation corpus: 5,712 speakers for training, 1,753 for evaluation, with N in 5, 10, 15 utterances per speaker. Anonymization uses Stream-Voice-Anon, a real-time neural system, and transcripts come from Whisper-medium (WER about 21%).
Aggregation helps, and more is better. Frame-level is the strongest. In the A-A condition (both enrollment and target anonymized), end-to-end frame aggregation gives a 39.30% relative EER reduction at N=15 over the WavLM-ECAPA baseline. With the stronger semi-informed attacker, frame aggregation pushes the equal error rate (EER, lower is sharper) down to 6.96% at N=15 (the lazy setting sits at 22.84%).
| Setting (A-A, N=15) | EER (%) |
| WavLM-ECAPA baseline (mean pooling, lazy) | 37.59 |
| Frame-level end-to-end (lazy) | 22.84 |
| Frame-level end-to-end (semi-informed) | 6.96 |
Multimodal beats unimodal, and the most counterintuitive result is that after anonymization, text alone identifies the speaker better than audio alone. At N=15, A-A, text-only reaches 28.20% EER versus 37.59% for the audio-only baseline. Anonymization changes the voiceprint, not the words, so the linguistic identity signal survives intact.
| Modality (A-A, N=15, lazy) | EER (%) |
| Audio-only | 37.59 |
| Text-only | 28.20 |
| Audio + Prosody (concat) | 25.20 |
| Audio + Text (0.5A + 0.5T) | 22.63 |
Equal-weight audio+text (0.5A+0.5T) gives the lowest A-A EER across all N, cutting 37.59 down to 22.63. The abstract's claim that five anonymized utterances of audio+text cut EER by more than 15% over audio-only maps to N=5: 43.77 down to 37.18, a 15.06% drop. RJCA goes further in Table IV, reaching the lowest A-A EER across all N, down to 27.23 at N=15.
For anyone working on speech privacy or security, the takeaway is direct: current anonymization evaluations overstate protection because they test single utterances with audio-only attackers. Real attackers (a full podcast back catalogue, call-center recordings, an interview series) hold multiple utterances and will fold transcripts and prosody into the attack.
The fact that text identifies better than audio is a design-level warning. The voiceprint is swappable; word choice and syntax are not. Hiding a speaker takes more than changing the acoustics; the linguistic style has to be obscured too. Any product relying on voice conversion for anonymization needs a threat model that spans multiple utterances and multiple modalities. This is not far from general engineering: any system that stores, transcribes, and re-distributes user voice (meetings, call QA, podcasts) and treats voice-print anonymization as sufficient can be pierced by exactly this kind of attack.
The limitations the authors flag are the substantive ones. First, the experiments use utterance-level anonymization (each clip anonymized independently) and never test speaker-level anonymization, where the same pseudo-speaker is assigned to one person across all their utterances. Speaker-level is plausibly the more realistic and harder-to-attack deployment, and the results could shift considerably; the paper does not go there.
Second, the text representations come from Whisper transcripts of anonymized audio (WER about 21%), and the impact of transcription errors is not isolated. Part of the text signal may be the recognizer's own quirks rather than clean linguistic identity, so that channel's true strength is somewhat overstated.
Third, only one anonymization system (Stream-Voice-Anon), one language (English), and one speaking style (telephone conversation) are tested. Whether the conclusions transfer to other anonymizers or to Mandarin is not established.
One more caveat from reading, not from the authors: the 6.96% headline number requires the full stack of attacker-favorable conditions, semi-informed, frame-level, end-to-end, A-A, N=15. It is real but narrow, and should not be quoted as proof that anonymization is broken in general.