MECT: Mixture of Experts with CNN-Transformer Network for Speaker verification
Yu Zheng, Jinghan Peng, ChangHao Zhang, Jian Liu, Weiqiang Wang
cs.SD, cs.AI, eess.AS
2026-09-21
Ant's MECT-B2 puts frame-level 4-expert MoE in a CNN-Transformer speaker net, hitting 0.012/0.026/0.048 VoxCeleb1 minDCF at 9.57M params, with 100ms causal chunks near offline EER.
Speaker verification asks a binary question: are these two clips the same person. Backbones have moved from ECAPA-TDNN and ResNet variants to CNN-Transformer hybrids such as CAM++ and ReDimNet, then to fine-tuning 587M-class self-supervised models like w2v-BERT 2.0. Mixture-of-Experts is standard in large language models. In speaker verification it had only shown up when fine-tuning pretrained models, as a way to mix layer-wise features. Nobody had put MoE into a fully supervised speaker net trained from scratch.
A second constraint is operational. Access control and call-center scoring often cannot wait for the full utterance. They need embeddings on 100ms chunks. Non-causal models fall apart when you slice that short.
MECT, from Ant Group's Machine Intelligence team, inserts MoE into a CNN-Transformer backbone and adds a causal-retraining recipe for streaming. The code lives in AntSpeaker.
The backbone follows ReDimNet's reshape-dimension trick: a CNN reads local spectrogram structure, a Transformer reads time, and a reshape mixes frequency and channel axes. Four block-level changes sit on top of that.
The residual add happens after BatchNorm and ReLU, not before. Position encoding is a two-layer 1D convolution. Temporal downsampling and channel expansion live inside the ResNet block's own strides, not as an extra conv in front. Blocks connect with a plain shortcut. ReDimNet's dense connections added compute without helping accuracy, so they were dropped.
MoE replaces the first linear projection in the Transformer feed-forward block. Two axes, four variants: route per frame or per utterance; mix all experts (Dense) or keep Top-K (Sparse).
Frame-level Dense is the simple version. Each frame goes through a linear gate, Softmax produces N weights, every expert runs, outputs are summed. Frame-level Sparse uses the same gate and keeps only Top-K. Utterance-level variants mean-pool the whole clip first, then emit one set of weights shared by every frame.
The net has four stages with 2/4/5/3 blocks, shrinking frequency from 80 bins to 10. Size is set by channel C and ResBlocks per block M. A models use C=16, B models C=32; the "1" variants use M=2, the "2" variants M=3. Transformer width stays at 64. Features are 80-dim log Mel filterbanks. Pooling is ASTP. Loss is SphereFace2. Embeddings are 192-d.
Streaming has three recipes. M1 averages chunk embeddings. M2 concatenates features then pools. M3 converts the net to causal form (left-only conv padding, causal attention masks) and retrains. At inference it caches conv receptive fields, attention KV, and ASTP's frame features plus pre-softmax logits.
Ablations train on VoxCeleb2 and test on VoxCeleb1. The no-MoE baseline has 9.40M parameters, 0.84 average EER and 0.064 minDCF. Frame-level Dense with 4 experts is the winner: 0.80 EER, 0.059 minDCF, a 4.7% and 7.8% relative drop, at 9.57M parameters. Expert count is brittle. Two experts land at 0.86 average EER, worse than no MoE. Five experts send Vox1-O from 0.27 back to 0.35. On Vox1-H EER, 4-expert Dense and no-MoE both sit at 0.85.
The public comparison uses that 4-expert Dense setup.
| Method | Params | Vox1-O EER / minDCF | Vox1-E | Vox1-H |
| ReDimNet2-B6 | 10.45M | 0.29 / 0.027 | 0.52 / 0.050 | 0.99 / 0.104 |
| MECT-B2 | 9.57M | 0.27 / 0.024 | 0.46 / 0.048 | 0.85 / 0.082 |
| W2V-BERT 2.0 (plus VoxBlink2) | 587M | 0.14 / 0.020 | 0.31 / 0.032 | 0.73 / 0.071 |
| MECT-B2 (plus VoxBlink2 + AS-Norm) | 9.57M | 0.22 / 0.012 | 0.28 / 0.026 | 0.52 / 0.048 |
Against ReDimNet2-B6, MECT-B2 cuts Vox1-H EER from 0.99 to 0.85 (14% relative) and minDCF from 0.104 to 0.082 (21% relative). After VoxBlink2 training plus AS-Norm and QMF calibration, minDCF reaches 0.012, 0.026, 0.048, which the paper calls state of the art on VoxCeleb1. Under the same calibration, W2V-BERT 2.0 still wins Vox1-O EER (0.12 vs 0.22). MECT wins minDCF.
On Chinese CN-Celeb they switch to frame-level Sparse Top-4 with 8 experts. MECT-B2 gets 4.87 EER and 0.297 minDCF. ReDimNet2-B6 under the same recipe is 5.22 / 0.321; dropping MoE is 5.01 / 0.303. That is 6.7% and 7.5% relative over ReDimNet2.
Streaming is the sharper result. At 100ms chunks, non-causal M1 jumps from 0.27 to 12.34 EER on Vox1-O; M2 jumps to 7.50. One-second chunks still work (M2 at 0.39). After causal retraining, the offline baseline is 0.38 / 0.59 / 1.05, and 100ms chunks stay at 0.38 / 0.60 / 1.06.
Routing in the first block lines up with phonemes. Fricatives and affricates (such as /s, z/) go mostly to expert E2, weights 0.65 to 0.88. Vowels go mostly to E0, weights 0.40 to 0.55. No phoneme labels were used.
Two takeaways for anyone shipping speaker embeddings.
The small model can contest the calibrated scores of a 587M pretrained net. 9.57M parameters and 14.43 GMACs on 2-second audio (backbone only) is about 1/60th the size of W2V-BERT 2.0. With VoxBlink2 and score calibration, minDCF comes out ahead. Compute is not cheaper: ReDimNet2-B6 is 13.11 GMACs, MECT-B2 is slightly higher. The gain is error rate.
The other takeaway is 100ms chunks. Door access and mid-call screening cannot wait for a full sentence. Causal retraining stops the short-chunk collapse. The bill is offline EER rising from 0.27 to 0.38.
The MoE itself is incremental: 4.7% average EER, 0.17M extra parameters. Dense four-expert inference still runs every expert. There is no FLOP saving, only gated capacity per frame. VoxCeleb prefers Dense, CN-Celeb prefers Sparse, so a new dataset may need that ablation again.
Four sizes from A1 (3.78M) to B2 (9.57M) cover different on-device budgets. Code is public.
The "state of the art" line is minDCF. Calibrated MECT-B2 still trails W2V-BERT 2.0 badly on Vox1-O EER (0.22 vs 0.12). The paper does not say which metric maps to the product.
MoE does not move Vox1-H EER in the ablation. Going from 4 experts to 5 undoes the Vox1-O gain. That is a narrow setting, not a plateau.
Causal streaming reports chunk accuracy, not wall-clock latency. "100ms" is audio length, not end-to-end time. KV and ASTP caches were left uncapped, so long-session memory was not measured.
The phoneme story is a correlation in the first block, aligned with Allosaurus. It is not causal. Later blocks are unexplained.