A foundation model of vision, audition, and language for in-silico neuroscience
Stéphane d'Ascoli, Jérémy Rapin, Yohann Benchetrit, Teon Brooks, Katelyn Begany, Joséphine Raugel, Hubert Banville, Jean-Rémi King
q-bio.NC, cs.LG
2026-05-06
TRIBE v2 trains a tri-modal Transformer on 1,118 hours of fMRI from 720 people, beating a matched linear FIR and recovering FFA, PPA and language maps zero-shot.
Cognitive neuroscience still runs as specialist models. Motion is mapped to V5, faces to the fusiform gyrus, written words to the visual word-form area, and each paradigm gets its own linear encoder. Change the subject, the task, or the modality and the model is retrained. The linear assumption is shaky on its own: network latents and brain responses are not linearly related. Labs do not have a single model they can query on a new person or a new protocol before booking the scanner.
TRIBE v2 treats whole-brain fMRI as high-dimensional regression. Inputs are the video the subject saw, the audio they heard, and the transcript of that audio. Three frozen pretrained models supply features:
Each stream is projected to 384-d and concatenated to 1,152-d. An 8-layer, 8-head Transformer reads 100-second windows, then adaptive pooling downsamples to 1 Hz to match BOLD. A subject-conditioned linear layer maps latents onto 20,484 fsaverage5 vertices plus 8,802 subcortical voxels from eight Harvard-Oxford regions.
Two training tricks keep the model usable when data is missing. Modality dropout zeros each stream with probability 0.3, resampling so at least one remains, which is how silent film or a podcast still produces a prediction. Subject dropout with probability 0.1 bypasses the subject layer and uses a shared "unseen subject" linear map, which is how group-average zero-shot prediction works. The loss is mean squared error on BOLD. AdamW, at most 15 epochs, one 32 GB V100 for a day. Caching features takes 128 of the same GPUs for 24 hours.
The baseline is not ordinary ridge. Deep FIR uses the same frozen embeddings, a linear convolution spanning 9 TRs with a 5-second offset, trained with SGD. Features and optimization scale are matched; the gap is the Transformer mixing time.
Training uses 25 people and 452 hours of deep data (Courtois NeuroMod, BoldMoments, Lebel2023, Wen2017). Testing uses 695 people and 666 hours of wide data (NNDb, LPP, Narratives, HCP). Combined: 720 people, 1,118 hours.
TRIBE v2 beats Deep FIR on every dataset (paired t-test across subjects, q(FDR)<10^{-4}). On Courtois NeuroMod, encoding score rises log-linearly with hours per subject and does not plateau. An earlier iteration won Algonauts 2025 among 263 teams, with mean 0.2146±0.0312 against 0.2096 for second place.
Zero-shot, the model's group-average prediction sits closer to the cohort mean than most individual recordings. On HCP, the only 7T set and the one with the most stimulus repeats, Rgroup is near 0.4, about twice the median subject's group-predictivity. Fine-tuning one epoch on at most one hour per new subject yields a two- to four-fold gain over a linear encoder trained from scratch on that person.
The harder test replays IBC functional localizers the model never trained on. Images flash for 1 second every 8 seconds; predicted activity in the ventral stream peaks 5 seconds after onset, matching hemodynamic lag. Predicted contrast maps spatially correlate with real IBC maps: faces in FFA, places in PPA, bodies in EBA, characters in VWFA. Language trials synthesize speech from text and feed both streams. A 10-word sentence starts in primary auditory cortex at t=3 s and spreads through the language network. Speech versus non-speech picks associative auditory cortex A5, and more weakly STS and Broca's area 45. Emotional versus physical pain picks TPJ and MTG. Sentences versus word lists, and complex versus simple sentences, produce the expected left-hemisphere bias and a semantic-versus-syntactic split.
ICA on the unseen-subject layer yields five components that line up with primary auditory cortex, the language network, motion, the default-mode network, and the visual system, and they correlate spatially with NeuroSynth maps for those keywords. Unimodal ablations rank video above audio above text, with complementary cortical maps. Around the temporo-parieto-occipital junction, the multimodal model beats the best unimodal encoder by up to 50%. Subcortical scores are two to three times lower than cortical scores, still above chance in most regions.
For an fMRI lab, this is closer to a digital stand-in that can be queried before scanning: new stimuli and protocols can be piloted in the model first. It is an encoding model. Direction of travel is stimulus to whole brain; it does not reconstruct images or words from neural activity. Unlike hyperalignment, it does not need every subject to see the same stimulus set. Code and weights are public.
The step from v1 to v2 is mostly scale. v1 predicted 1,000 parcel means on four subjects and one task. v2 predicts 20k vertices plus subcortex, trains on 25 people and four tasks, and adds the unseen-subject head, which is what makes IBC-style controlled experiments possible. The linear baseline already uses the same frontier embeddings; the extra points come from the Transformer mixing time. First place at Algonauts is 0.005 above second. A competition score is not the same as a working digital brain.
The authors list the hard constraints. fMRI cannot resolve millisecond spiking. Inputs cover vision, audition, and semantics, and skip olfaction, balance, and somatosensation. The brain is treated as a passive observer, not an agent that acts. The data are healthy adults; development and clinical pathology are out of scope.
A few method choices also clip the claim. Spatial pooling of video patches is expected to hurt retinotopic cortex, and the paper does not report how much. IBC language stimuli were translated from French to English, run through TTS, then aligned with WhisperX, so they are not the original sounds. In-silico recovery is given as spatial correlation and a qualitative match; the text does not report an r for each contrast. Subcortical scores being two to three times weaker is a reminder that "whole brain" is still uneven. Training is almost entirely natural movies and podcasts. Matching IBC is encouraging. There is no evidence yet for decision-making, motor tasks, or clinical protocols.