Decoding Perceived Speech from MEG Using CLIP-style Architecture
Ilia Semenkov · hf · 2026-08-07
This study introduces an improved deep learning architecture to retrieve and decode perceived speech from non-invasive magnetoencephalographic (MEG) recordings.
Core Methodology & Improvements
- Architecture: Employs a CLIP-style objective to contrast MEG signals against wav2vec 2.0 audio embeddings.
- Physical Priors: Replaces flattened spatial attention with spherical harmonics defined on the 3D MEG helmet geometry, reducing subject-specific representation branches from 270 to 25.
- Anti-shortcut Training: Ocular and cardiac artifacts are removed prior to training to mitigate the risk of stimulus-locked shortcuts.
Results & Interpretability
- Performance: Achieves 39.75% Top-1 accuracy among 1005 candidates on the MEG-MASC dataset, using approximately 20x fewer decoder parameters.
- Neuroscientific Mapping: Model weights successfully map to brain source space, recovering generators consistent with the human speech-perception network. Left-lateralized branches were found to carry higher-frequency rhythmic components.
- Driving Features: MEG occlusion reveals that 15 stimulus features contribute most to retrieval, notably silence, sound intensity, vowels, and acoustic onsets. Furthermore, coherent narrative speech contains significantly more recoverable information than random word lists.
More from Research
- Google Open-Sources WeatherNext 2: Generates 15-Day Forecasts in Under a Minute — aigclink · 2026-08-07
- Google Open-Sources WeatherNext 2: Generates 15-Day Forecasts on a Single TPU in Under a Minute — aigclink · 2026-08-07
- UT Nuremberg Opens PhD Position on VLA Models and 3D Geometry — y_m_asano · 2026-08-07
- Stanford and Arc Institute Use AI to Design Working Bacteria-Killing Viruses — The Decoder · 2026-08-07
- RL Training Collapses at Step 8: Dev Implements LoRA to Fix GPU Sync — mervenoyann · 2026-08-07
- Noether: Open-Source Tool to Automatically Prove Math Properties of JAX Code in Lean — jonkhler · 2026-08-07