Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrieval
Ilia Semenkov, Daria Kleeva, Ivan Dakhtin, Zarina Maksudova, Alex Ossadtchi
cs.LG, cs.SD, q-bio.NC
2026-08-03
A 20x smaller interpretable MEG speech decoder keeps Top-1 at 39.75% on MEG-MASC, with weights that localize to auditory cortex and reveal which speech features drive retrieval.
In 2023, Défossez et al. (Nature Machine Intelligence) showed a deep net can pick, from more than a thousand 3-second clips, the one a person is listening to, using only non-invasive MEG of their cortex. The method encodes brain and audio into vectors and trains a CLIP-style contrastive objective, pulling matched pairs together and pushing the rest apart, with wav2vec 2.0's 768-dim representation on the audio side.
The catch is that the network is a black box. Its weights map to no electrophysiological quantity, not a cortical site, a rhythm, or a time course, and nothing says which speech properties actually drive the retrieval. The model predicts, but you cannot tell what it is predicting. Worse, eye movements track attended speech and heart rate tracks narrative tension, so these peripheral signals are themselves stimulus-locked, and the network has every reason to free-ride on them while you think it is reading cortex.
This is a paper about turning a working black box into something readable, and about closing the loophole that what you are reading might be an artifact rather than cortex.
Four changes to the Défossez architecture, each aimed at making the front-end weights readable as physiological quantities:
A 3-second window becomes a 768-dim embedding, contrasted against the wav2vec features of its audio.
Main numbers, averaged over six independently trained solutions:
| Configuration | Top-1 | Top-10 | Params |
| Main (K=25, 2 blocks) | 39.75% | 70.40% | 486,619 |
| Défossez et al. (reference) | 41.3% | 70.7% | roughly 20x more |
| Own grid, K=270, 5 blocks | -3.60 pp | -3.14 pp | 14.8x larger |
| LinearDR-12 (target to 12 dims) | 39.95% | 70.54% | 427,651 |
The authors are explicit that the cross-study comparison is not directly rankable: candidate pools differ (1005 here vs 1363/1464), windows are fixed-stride here but word-onset-aligned there, and only this pipeline removes ocular and cardiac artifacts. So read 39.75% next to their own controlled ablation, where the bigger model is both larger and worse. Swapping spherical harmonics for 2D Fourier costs about a point, and dropping the temporal filter costs about a point; both exceed the 0.34-point spread across random seeds.
Cortical mapping: RAP-MUSIC dipole fitting on the front-end weights lands in bilateral auditory cortex and the frontal lobe, with distributed maps clustering along the superior and middle temporal gyri and peri-Sylvian cortex. Left-lateralized branches carry higher-frequency rhythms absent on the right, matching the known speech-perception network.
The occlusion analysis is the centerpiece. Of 19 acoustic, phonetic, and linguistic features, 15 contribute; the largest are silence (delta-r 75.62), high loudness (60.77), vowels (38.70), and strong acoustic onsets (36.19), all positive in every one of the 27 participants at p<10^-4.
The counterintuitive result is random word lists. Substituting narrative MEG into a word-list interval improves retrieval (delta-r -17.77): brain activity during randomly ordered words carries less recoverable information than during coherent narrative. Pseudowords give a positive effect (an N400-like violation of an established expectation), whereas a random word list removes the context that generates expectations, so the decoder loses the predictive structure that supports tracking.
Compression: the 768-dim wav2vec target shrinks to about 12 learned dimensions with no accuracy loss (power-driven PCA degrades earlier), but global pooling of the within-segment trajectory drops retrieval toward chance. The target can be low-rank; the temporal structure cannot. Longer is better: Top-1 climbs from 14.37% at 1.5 s to 62.20% at 5 s.
Interpretability does not have to be paid for in accuracy. The model is smaller, equally accurate, and returns weights that read directly as cortical sources, breaking the default assumption that only a black box can gain points. The paired occlusion recipe (paired MEG substitution plus sign-flip permutation testing) is a reusable audit for any brain-decoding model, answering two questions: is it reading cortex or a peripheral artifact, and which stimulus properties does it actually use. Stripping ocular and cardiac signals before training is a discipline more decoding papers should adopt.
The negative word-list result is the sharpest finding in the paper: the decoder rides on the predictive structure of coherent narrative, not on anomaly or N400-style surprise. That narrows what speech decoding means here.
For machine learning more broadly: the K=10-25 plateau and the 12-dim target show that task-relevant M/EEG lives in a low-rank subspace even when the target is a high-dimensional contrastive embedding, extending the EEGNet-era lesson that small filters suffice into contrastive alignment with wav2vec.
Ready to use? Not yet. This is non-invasive retrieval of perceived (not imagined) speech, not an imagined-speech brain-computer interface, and it is not clinical.
The authors state several firm limits. There is a single corpus (MEG-MASC, 27 participants); generalization across listeners holds only conditional on the fixed held-out audio, and whether the feature-use pattern replicates for new narratives, corpora, or architectures is unverified, with the cross-seed check covering only initialization. The front-end reading assumes linear space-time separable processing in layer 1 and can miss space-time-inseparable cortical waves. Occlusion does not isolate a strictly independent causal contribution per feature: annotations overlap and masks differ in duration, so the rank contrasts mean only that the decoder uses feature-present-versus-absent MEG information, not a calibrated ranking of neural encoding strength.
Cortical mapping is also loose: because defacing truncated the structural scans, FreeSurfer reconstruction succeeded for only 6 of 27 participants, so everyone is mapped through one template brain and one generic forward model, and anatomical variability spreads the maps.
When reading the numbers, keep in mind that the authors hedge the cross-study comparison themselves, so 39.75% belongs next to their own ablation; and the largest occlusion effects (silence, loudness) are partly inflated by mask duration, which they flag, so silence is not necessarily the most important feature.