Decoding Perceived Speech from MEG Using CLIP-style Architecture
Ilia Semenkov · hf · 2026-08-07
This study introduces an improved deep learning architecture to retrieve and decode perceived speech from non-invasive magnetoencephalographic (MEG) recordings.
Core Methodology & Improvements
- Architecture: Employs a CLIP-style objective to contrast MEG signals against wav2vec 2.0 audio embeddings.
- Physical Priors: Replaces flattened spatial attention with spherical harmonics defined on the 3D MEG helmet geometry, reducing subject-specific representation branches from 270 to 25.
- Anti-shortcut Training: Ocular and cardiac artifacts are removed prior to training to mitigate the risk of stimulus-locked shortcuts.
Results & Interpretability
- Performance: Achieves 39.75% Top-1 accuracy among 1005 candidates on the MEG-MASC dataset, using approximately 20x fewer decoder parameters.
- Neuroscientific Mapping: Model weights successfully map to brain source space, recovering generators consistent with the human speech-perception network. Left-lateralized branches were found to carry higher-frequency rhythmic components.
- Driving Features: MEG occlusion reveals that 15 stimulus features contribute most to retrieval, notably silence, sound intensity, vowels, and acoustic onsets. Furthermore, coherent narrative speech contains significantly more recoverable information than random word lists.
More from Research
- AI will be superhuman at science regardless, and Europe can't opt out — teortaxesTex · 2026-10-16
- SERO augments long-horizon agentic RL with executable oracle continuations — Shanghai-AI-Laboratory · 2026-10-16
- Tsinghua and Amap propose OmniVerifier-C1, a graded verifier beating binary judges — Bowei Liu · 2026-10-16
- Alibaba open-sources MAI-Gym, an all-in-one GUI agent eval and RL training platform — Tongyi-MAI · 2026-10-16
- Peking University's SimForcing distills simulation motion priors into robot world models — PekingUniversity · 2026-10-16
- CoWorld Reasoner uses an evolving video world model as a visual reasoning engine — ModalityDance · 2026-10-16